4-Langchain Series-Getting Started With RAG Pipeline Using Langchain Chromadb And FAISS

hello guys we are going to continue the Lang chain series and in this specific videos and in the series of upcoming videos we are going to discuss about rag pipeline now rag uh the full form is retrieval augmented generation one of the very important use cases you may probably solving when you specifically use llm models and most of the use cases right now in companies are demanding this kind of skill sets you know where you can actually develop rag let's say you have a set of documents and you should be able to query that specific documents and get the result quickly if you have different kind of files let it be a readme file a text file a different data source file you should be able to read it convert that into a vectors and able to retrieve any queries specifically from those kind of data sources so uh as I said uh we will be implementing this completely from scratch from basic to Advanced in this specific videos we'll understand the entire architecture and then we will do a lot of practical intuitions okay so uh in any rag pipeline these are the major components that will probably exist so the first component is specifically called as load data source now whenever we say load data source initially we may have see rag is all about quering from a def different different data source alog together right so we may be having files that may be having PDF it can be a MD file readme file it can be a Excel file it can be a txt file it can be a data databas file it can be different different files Al together right and in the first first step that we see over here it is basically called as load Source data this step is also called as data inje okay data inje now in Lang chain the most amazing thing is that it definitely have a lot of different kind of data injection tools which will be able to load the data in various manner so in our current video we are going to implement each and every of this component from data inje till the query Vector store okay then after ingesting the data you can probably do load you can transform and you can embid okay we will discuss about this what exactly is load transform and embade specifically when we do loading that is nothing but we are reading from a specific data source then we perform some type of feature engineering over here if we want like in transform stage here the data the complete data will be broken into smaller chunks okay now why do we divide this data into smaller chunks it is very much important to understand because whatever llm models we specifically use right is definitely has some kind of context size right so based on this context St it is always a good good way to basically convert this entire data which can be of many number of PDFs it can be U many number of pages in this specific specific PDF we will try to divide this particular thing into chunks of data so this is also what we'll going to see in the practical way then finally we go ahead with embeddings embeddings basically means how we can convert all these chunks into vectors okay how we can actually use vectors in order to probably convert this chunks right and finally all these vectors will be further stored in some kind of vector stored database okay so it can be a dat database over here and the main reason of this specific database is that we will be able to query this particular database in an efficient manner right with respect to any query that we have right so whatever vectors are basically stored if I have a query if I hit the query on this particular database I should be able to get the result based on the context of the query so this is the entire rag pipeline that we specifically use in the upcoming series of videos we are going to use different different vector databases in different different clouds also okay so right now we will try to implement all these things that you can actually see in front of you and we will follow this entire architecture so let me quickly open my VSS code and once I probably start my vs code I will be um I will continue with respect to the same uh thing that I have actually used right all the projects that I've actually created as I said that all the things will be given in the description of this particular video with respect to the GitHub link okay now now I've have created a folder which is called as rag inside this I will create one ipynb file okay initially we'll go with ipynb file and then later on we'll try to create an end to end project okay so here I will say uh simpler rag. iynb since we are doing it completely from Basics it is always good that we try to do it completely from basic itself right so U this is the thing so first of all I will go ahead and see whether in my terminal whether I have that I kernel installed or not okay so I will go ahead and write pip install ipy kernel IPI kernel is specifically used to install the Jupiter notebook kernels itself right so once I probably install this you'll be able to see that this installation will take place and I think it is already satisfied so it is good enough we can probably go ahead and start the coding over here then I will go ahead and select my kernel and it is nothing but 3.1 10. zero right this is what is the environment that I have created now if I go ahead and execute something so it is giving me some kind of error okay so perfect now what I'm actually going to do uh I can see that there is some kind of message over here I will just try to uh disable this I've already uninstalled uh this extension so that it will not give us any disturbance so let me do one thing let me just open over here for you and here I will open this particular folder and then I will go ahead and close this okay close window okay and now let me quickly open the vs code again and here is my entire project we will just try to execute it once again okay perfect so that so that I don't have any disturbance with respect to that okay now perfect now this is done now let's go ahead and start our coding initially I will try to show you multiple data injection steps okay so data injection basically means let's say I want to read from a text file I want to read from a PDF file I want to read from a web web page how can I actually do it okay so all those things we'll start to discuss so first of all I will go ahead and import from Lang chain unor Community do document loaders now document loaders has all the techniques specifically if you want to load uh let's say a PDF you want to load a Excel file you want to load a txt file so over here all the libraries will be present so from chinor community. doent loaders until then I will just close this terminal because I will not require over here then the first one that I'm actually going to discuss about text loader itself now over here what I'm actually going to do I'm going to create a loader is equal to text loader and after creating this text loader here I'm going to give a text file okay speed. txt so we'll just go ahead and see whether we have the speech. txt file or not no I don't have it so let me go ahead and create one speech. txt file okay and uh with respect to this particular file what I am actually going to do I'm going to put some content so I have already have that content let me just quickly copy and paste it this is one of the most famous speech um that is available in some history it has been taken over here so I just Googled it and I bought this particular speech over here and saved it in speech. txt okay uh then what I will do I will go over here and read the speech. txt okay now just by using this text loader what will happen is that you'll be able to read the speech. txt then I will use loader. load and here I'm going to specifically convert this entire thing into text documents okay once I write loader. load that basically means it is going to just convert this into a text documents and this finally you'll be able to see my test documents and I will go ahead and execute it so here you'll be able to see that once it is reading the specific speech it has the entire text document now it is becoming so easy just to read the txt file okay now the next thing that I'm going to specifically do is that uh since I'm also going to use open a keys or um I'm also going to use Ama embeddings or AMA model so what I can do is that I will just go ahead and import OS um the next thing that I will go ahead and do is lo load uncore Dov okay so load. EnV so for this I'm also going to import from. EnV uh import load. EnV okay now why I'm doing this so that I will be able to call all my environment variables and in one of the environment variables I also have my open API key so now I will go ahead and write OS do environment and here I'm going to use my opencore a sorry open aior API underscore key and here I'm just going to use os. get EnV and here again I'm going to use my open API key so this is basically going to call the open API key from the environment variable now a very important step alog together over here guys now is the main term that we will be starting to uh see some more data injection techniques so one more data injection technique is directly reading from the web based web based uh loader you can basically say okay and for this again I will be importing understand any document loader will be available in this specific Library itself and instead of writing text loader we will be using webbased loader okay now along with this I will also import bs4 that is beautiful soup 4 so for that also what I will do I will just import over here in my requirement. txt bs4 so that whenever I execute this it should be able to run okay now in order to do something like this load chunk and index the content of the page HTML page right or any web page so here I will create a loader and this will be equal to web P loader and here I'm going to give two important parameters one is the web path so webcore paths okay is equal to and here I will give the URL so let me just take one URL from the GitHub IO page that is also available in the documentation of Lang chin so this is what is the page over here um what I will do is that quickly I will open a browser and show it to you how this page looks like okay so if I execute this over here you'll be able to see that just a second okay perfect so here you you are able to see that uh just a second this see is this the same page yes it is the same page and I will be opening this okay perfect so this is the page that you will be able to see over here and I want to probably read this entire page and uh use it as a rag system itself right so in order to read this entire page I can just take this URL and use this web loader Library so here if you see I'm using this web based loader library and I'm giving the first parameter that is the web path okay the second parameter that we can specifically give is our arguments right so here I will rate BS doore KW R GS okay so this is the second parameter I'm just seeing the documentation and we will give the next argument in the form of dictionaries the first is parscore only okay and then here I'm basically going to use my bs4 that is a beautiful Soup For Soup trainer okay soup strainer soup strainer okay yeah soup strainer so this is the first uh parameter that I will be specifically giving and understand why this is required because here the soup strainer and since we are using beautiful soup over here we really need to give the class is that it read it needs to read from that particular page okay and here what all classes are there let's go ahead and see this okay so if I probably see over here and if I do just inspect element okay so here you can see post title is there then you have something like post content where all the content is basically available so let us take this one as my first one okay so I will just go ahead and execute this and here you'll be able to see that uh this will basically be my post title and here you have the postore content and third one that I also want to take is postore header because header will also have some information over here right so this is done you'll be able to see that it looks absolutely fine and here with respect to this we have also created this entire loader um and here I will use a Comm up so that this is basically my argument over here right right so once we do this uh then let's see whether it will execute or not uh bs4 is not found as I said that bs4 is not there so let me do one thing quickly let me open my command prompt and do the PIP install requirement. txt okay so here I will go ahead and write pip install minus r requirement. txt and execute it over here you'll be able to see now bs4 will also get installed and once it is getting installed it is completed it is good to go now if we go ahead and execute it I think it should work perfect now loader is there now what we are going to do over here is that again write loader. load and finally I will be able to get in the form of text documents textor documents okay documents and here is my text document so perhaps you meant uh this one let's see um so guys there was one one error we need to give this URL in the form of duple so let this get closed and then I will write comma over here now I think it should be working let's see yes perfectly it is working right now and if you go ahead and see your text documents again you will be able to get all the information from that particular web page in the form of URL okay or in the form of documents so this is perfect uh you able to get every information over here and uh with respect to this you will be able to see that yeah you have all the information and you can also see the page content and everything looks fine so these are some of the ways how you can specifically make sure that uh you probably get all the content from a page itself one more thing uh this is not underscore this is Dash so now I think you'll be able to see more content okay so let me just open this so here you'll be able to see all the content itself so uh this is one more way one more way is directly to read from the PDF itself so what I will do is that I will quickly create a PDF okay so let me do one thing and I will upload a PDF over here okay so here I have created this attention. PDF this is nothing but attention is all you need uh now uh using document loader you can also directly read completely from the PDF itself okay and how to do that see still we are in the data injection phase and how do we do this we'll also see that so from uh the same library that we'll be using again we have to focus on document loader and here I will copy and paste it okay um and this is basically my PDF reader one more Library dependency will be there okay so we will try to install that also so here I'm going to write Pi PDF all loader okay and again I will be using loader is equal to Pi PDF loader Pi PDF loader and I have to give my PDF name right so PDF name over here specifically is nothing but attention. PDF so it is in the same same folder itself so here I will write ATT ion.pdf okay now I think I should be getting an error if I don't get an error I will definitely install yeah I'll install it now see pip install Pi PDF so here I will make sure that I will write requirement. txt and I will write Pi PDF okay so all these things you really need to take care of because there will definitely be dependencies because and understand rag is one very important thing that really needs to get created in the form of application so that is the reason I'm creating this completely from scratch so P install minus r requirement. XT so this is also done I think and Pi PDF is also installed perfect then the next step again I will go ahead and execute this and but here also the same step will be required loader. load and this will basically be my documents and I will execute this and once I probably execute this this is my docs okay perfect so here you'll be able to see my entire PDF has been read and it is in front of you now the next step now see this load data source part is basically done we know how to load it okay now we will move towards this transform okay then we will go ahead towards embid okay so the first part of load I've actually done with respect to PDF web based with respect to text file if you go ahead and check out the langin documentation this there are still more amazing ways to do for Excel file for readme file different directories even for director let's say that you have lot of files in the directory itself many files and you can also load that so we'll try to see see in upcoming videos more about different different examples but here just to give an idea about load data source and load is actually done now let's go ahead and do this transform now transform is very much important Now understand guys this is your entire PDF documents right now you need to convert this PDF documents into chunks now how do you do that and again there are multiple ways and this ways entirely Falls inside the category of text splitter inside Lang chain so here we are going to write from Lang CH from Lang chain dot text splitter import recursive recursive character text splitter we'll try to split it with the with the help of uh text itself so here I'm going to create my text splitter is equal to recursive character text splitter and here I'm going to specifically use my chunk uncore size is equal to th000 uh I want this specific size to be the chunk and let's say the overlap chunk chunk overlap okay chunk overlap so here I will go ahead and write chunk underscore overlap is equal to 200 I'll keep this as 200 okay so once I do this now this text splitter will be uh responsible textor splitter dot split documents will be responsible in splitting all the specific documents that I give over here okay so this docs will be there and this will be my documents final documents that I will be able to see okay let's display this documents the top five documents till here okay uh and let's see this so here you'll be able to see provided proper attrition is Google by Grand permission best model this this this this everything is there attention. PDF right so all the information is basically there um and with respect to this you really want to see the more things more uh different different uh all the documents in short so you can also go ahead and see the entire documents over here okay done so these are all my documents over here Transformers everything is available over here now see this is my entire documents it has been divided into proper smaller chunks now we can take this chunks and we can convert that into vectors okay and that is what is giv you an idea over here right like how we have transformed We have basically taken this entire PDF document and we have divided that into chunks now it's a time that we understand how to probably convert this into vectors and for this we will also be using some different different Vector em uh Vector embedding techniques right one of the vector embedding techniques that I will be showing you is with respect to open AI so here now we will go ahead and probably write about vector embeddings and Vector store right Vector embeddings is a technique wherein we convert a text into vectors okay so how do we do that again with the help of open AI so here I'm going to write from linore Community dot embeddings import open AI open AI embeddings you can also use AMA embeddings it is up to you since if you don't have open API you can use AMA embeddings directly but the performance of openi embeddings is better far more better than AMA embeddings okay so here what I'm actually going to do next thing is that first of all uh we need to understand how to probably create vectors again to create vectors we can use open AI embeddings but after creating the vectors we also need to store in the vector store that is what it is right this Vector store is like kind of a database embeddings can be embeddings is a technique where you convert text into vectors but later those text needs to be stored in some kind of vector store right so for this reason we will be using something called as chroma DB so they are come there are couple of uh there are couple of vector databases that has been provided by langin itself one is chroma one is f um as we go ahead you know how to create this kind of vector database in the cloud also I will show it to you so from Lang chain Community I will be using Vector stores and here I'm going to import one is nothing but chroma uh so with respect to this we will go ahead and create my DB and here I'm going to write chroma do frore documents okay from documents and here I'm going to basically give my entire document okay entire documents and I'll not give the entire documents let's just give the first 20 documents itself because it'll take more time to create the embeddings right and here the embeddings that I'm going to specifically use is nothing but open embeddings once I execute this I think I'm going to get an error saying that chroma is not available I don't know whether I've installed chroma or not so here you can see pip in install chroma DB I really require chroma DB so again I'll go back to my requirement. txt the reason why I'm showing you all this completely from scratch so that you whatever error you face you should be able to fix it up okay so I'm going to delete this and let's go ahead and write pip install uh one more library that I will be using is nothing but F CPU okay fire CPU because f is also one type of chroma one type of vector database okay so minus r requirement. txt so let's install both the specific libraries it'll take some amount of time again it depends on your internet speed and how fast your system is but yes I think 5 CPU and chroma DB is the kind of uh Vector databases that we will be using one assignment I'll give you try to also use Lance Vector database from again seeing the lanon documentation you can actually do it okay now this is done once uh this will probably be done then we can go ahead and check by executing it whether it is working now or not but at the end of the day here what we are actually doing we have imported open embeddings the vector database that we are going to use this chroma then chroma from document I'm giving the entire document and using this embeddings it will be storing inside this particular Vector store now this DB we can store it in our local we can store it in the cloud wherever you specifically want okay so guys the installation has been completed now let's go ahead and execute this chroma part now here you'll be able to see that entire embedding will basically happen and again we are going to use the open a embedding for this and now this DB is nothing but our Vector database okay so if I probably consider Vector database and now all I have to do is that I can query anything with respect to this particular Vector database to retrieve any kind of result that I really want so I will create a query first of all so here let me go ahead and write my query so query is like I will go ahead and write who are the authors who are the authors of attention is all you need research paper okay so this is my query that I'm going to specifically ask let's see whether it'll be able to understand this or and it'll be able to give us the result so here I will go ahead and write DB dot similarity search now there are multiple option similarity search similarity search by Vector if you want you can also convert your data into vectors by using again the open Bings and you can query it from here but we are just going to use the similarity search since we are just going to use the query over here right so if I write db. similarity search of query so this will basically be my result okay now let's go ahead and see my result okay and let's execute this so here you'll be able to see that I'm able to see multiple information like four documents has been over here so let's take the first document and let's see how the result is so here I'm going to basically take zero and inside that the field name is nothing but page content so I'm going to basically write my page content now here you'll be able to see prop provided proper attribution is provided Google here by Grant permission and here you'll be able to see Google brain all the email IDs all the researchers names are actually available so that basically means it is able to Res uh it is able to retrieve the results of the all the scientists who are involved all the researchers who are involved in creating this particular paper um I can also do one more thing I can go ahead and write what is attention uh is all you need okay attention is all you need Okay so let's see what kind of results I will be able to get and understand this is entirely rag pipeline which is being able to come from the entire documentation and here you'll be able to see that result is also coming up uh let me do uh write something which is available in the research paper so let me open my research paper over here and let me ask some question directly okay from here you will be able to see see when I'm searching attention is all you need it is coming from here itself right so let me use something over here uh let me just write over here encoder is uh what is an attention function let me just go and search this okay let's see attention is something something some text is there over here so if I go ahead and execute it and here you'll be instead of performing a single attention function with the model Dimension values keys and all the information is basically coming up right and here is a very good result and this is directly coming from the vector database that is the most amazing thing over here right now you may be also thinking CH can we also use the F database fire Vector database that we have used right so let me just show you with respect to that also F and here I'm going to use the fire Vector database and this will also give you an idea how you can actually store that embeddings into F database itself and and we'll also do that so here I'm going to write from Lang chain Lang chain uncore Community dot Vector stores I'm going to import my FIS F and then I'm going to also use TB is equal to FIS Dot from documents the same thing right so from underscore documents and here I'm going to use my documents okay and let's say I go ahead and do the embedding for the first 20 documents and then here also I'm going to use my open embeddings and I think this is also my another DB so I will write it at DB1 which is my f database okay uh from documents okay no worries uh so here the spelling mistake was there and I think this will also work now DB1 is also ready and what I can do I can use the same thing and paste it over here okay and just write DB1 do similarity search and here you can see instead of performing a single attention everything is probably coming up so I've just shown you the example of both F and uh chroma database chroma Vector database as I said one assignment will be given to you as regarding Lance Vector database you can go and search it for that in the in the in the lanin documentation but here if I probably see with respect to this we have completed this load transform embed then we have created this Vector store and then we have queried that particular Vector store and retrieve the most similar results right so this is just a beginning of creating a rag pipeline now after this in the next video we also going to discuss about retrievers chains how we can convert this how based on context we can retrieve different different data all those things we'll be specifically discussing about so I hope you like this particular video this was it from my side I'll see you in the next video have a great day thank you and all take care bye-bye

Channel: Krish Naik

Views: 20,550

Keywords: yt:cc=on, langchain rag tutroials, vector databases tutorials, rag pipeline tutorials, end to end rag projects, retrieval augmented generation

Id: 9Thc6hRw2Gs

Length: 30min 21sec (1821 seconds)

Published: Sun Apr 07 2024