Announcing LlamaIndex Gen AI Playlist- Llamaindex Vs Langchain Framework

hello all my name is krishak and welcome to my YouTube channel so guys I'm starting a new playlist on an amazing framework which is called as Lama index now from past couple of months I've been focusing more on generative AI uploading amazing videos implementing multiple end to end projects with the help of llm models so we have discussed already about open AI llm models we have discussed about Frameworks like langin we have created a lot of end to end projects not only that we have also seen open source llm models like Lama 2 from meta we have also used hugging face libraries we have seen how to do the deployment and finally we have also implemented multiple end to-end projects with the help of Google gini pro and Google gini Pro Vision all these things we have specifically done now one more important framework that I'm going to discuss about is called as Lama index again in this playlist there will be a series of videos and again we will see that how we can use Lama index and even Lan framework and create some amazing endtoend projects and harness the power of the llm models all those things will be covered now in this video I'm going to discuss about two important things what is Lama Index right and the second thing which many people still have a lot of confusion like what is the exact difference between Lama index framework and Lang chain not only that we'll also discuss about one simple architecture of a project if you're combining Lama index and Lang chain how your project architecture will look like and specifically because many people think that Lama index is also used to create an end to end gen project it is uh langin is also used to create an end to end gen projects or generative AI projects yes it is used but specifically in those architecture where exactly is Lama index used because Lama index is very good at something which I will be discussing about it and Lang chain is super important and beneficial for Sim some different applications in a specific project so both these things will get covered now let me go ahead and share my screen uh just to keep a target for this particular video guys please let's hit like and keep the Target to th000 likes at least because all these videos are completely for free and as I said that I want to democratize the entire AI education so please do help me in that so here exactly it is like here is the Lama index page itself as I said two important things I'm going to discuss right what exactly is Lama index why it is super beneficial and the second thing is we'll compare the differences between Lama index and Lang chain and with respect to a project architecture where exactly is llama index used and where exactly Lang chain used everything I'll be discussing about it now first to First over here this diagram will actually help you to understand where does Lama index actually work whenever guys uh in your application many people have different custom data right let's say it be companies let it be uh it can be different different data itself data source itself like like YouTube apis PDF it can be Notions it can be SQL and this side you can probably see all the llm models now if you have specific custom data now let's see the definition Lama index is a simple flexible data framework for connecting custom data sources to large language models now to connect this entire data source to this large language models right that is where in those pipeline Lama index can be used beneficially see with the help of Lama index we can create an end to end project but its core important feature is connecting the custom data to the llm models now how do we connect this you know we take this entire data we perform something called as indexing using Lama index and then once we create those indexing right then we will be able to query from that index because it creates the necessary metadata itself right this entire Lama index library and then when we quiring anything we'll be able to get the response quickly right we'll still discuss about multiple points with various features what what is the difference between Lama index and Lang chain but here I hope you got an idea llama index is a simple flexible data framework for connecting custom data source to large language models okay some of the functionalities uh you'll be able to see we'll be able to create document Q&A data argumented chat B knowledge agents structured analysis many more things we'll be able to create and the same thing we also able to create with the help of Lang chain but what exactly is the difference and in the entire project architecture which pipeline we specifically use Lama index and that is where you'll be able to see over here in this three important steps we specifically use one is the data inje that basically means it provides you lot of libraries to connect with your external data of different different data sources like apis PDF document SQL Etc then after ingesting the data it helps you perform data indexing again store and index your data for different use cases integrate with Downstream Vector store and and database providers and the third step is with respect to query interface Lama index provides a pro uh query interface that accepts any input prompt over your data and returns a knowledge augmented response so from that index whenever you put any queries you'll be able to get a good response not only good quicker response it is quite fast so if I consider L index if you tell me Chris tomorrow if in a project if it someone tells you to use l index where's specifically you're going to use in this three steps right now once we get this response we can further connect this to any llm powered app or llm models itself and based on the prompt engineering that we do a prompt template we can do we get a specific response and there we can specifically use langin again I will be discussing as I go ahead so I hope you got an idea with respect to this whenever we talk about Lama Index this three kind of data you can easily connect to it one is unstructured data structured data and semi- structured data right it come it it it supports all the this three different types of data itself now let me quickly go ahead and talk about some of the important differences between Lama index and Lang chain now guys this differences is super beneficial why I'm telling you because in interviews in projects when you implement you should know what is the thing that you should really use and finally after discussing this differences can we use Jama index and Lang chain together in a specific llm powered app and that kind of architecture also I will discuss about so over here all the differences that you'll be seeing and remember guys this uh I have referred it from di dw. a website so this is another blogging website over there where they they put a lot of information regarding generative AI so again the references from here I've taken the screenshot from there and but I'll explain it from my way okay so over here based on the features here you'll be able to see here you have Lama index here you have Lang Lang chain as you know both of them are Frameworks right and these are specifically used with multiple llm models but if I talk about what is the primary focus of Lama index so here you'll be able to see intelligent search and data indexing along with retrieval that three points that I specifically discussed about in case of Lang chain it helps you build a wide range of gen applications right what all different different all different different functionalities J application will help you to create if I talk with respect to data handling Lama index helps you in ingesting structuring and accessing private or domain specific data right it will be able to help you do all these functionalities in case of Lang chain loading processing and indexing data for various use cases see here also you can do indexing but with the help of Lama Index this this indexing that we are specifically doing it is very much efficient over here right why I will show you when I probably develop a project in my upcoming videos upcoming series of videos okay now if I talk with respect to customization it offers tools for integrating private data into llms whereas in Lang chain you can see highly customizable it allows users to chain multiple tools and components right what is this multiple chain multiple tools and components I'll discuss about when I talk about the architecture in case of flexibility but here again we talking about some data and integrating that data with our llm right mostly with respect to Lama index if I talk with respect to flexibility specialized for efficient and fast search see this is what is very much amazing in this efficient and fast search the kind of indexing the kind of metadata that is created using Lama index actually helps us to query that data efficiently and with less response time right in case of Lang chin general purpose framework with more flexibility in application behavior let's see with respect to llm model which all llm models is being connect uh it can probably support so it connects to almost most of the llm pro providers like open AI anthropic hugging face uh AI 21 Labs right in case of Lang chain it supports 60 llm models so it is pretty much good it is more than when comp to the Llama index now you may be saying Krish I'm talking all good points about Lang chain what about llama index guys exact distinguish I'm trying to talk about the exact differences right and once I probably compare compare all these particular uh differences then you'll be able to understand as soon as you see the architecture so use cases here you can probably see best for application that require quick data lookup and retrieval again quick data lookup and interval suitable for application that require complex interaction like chat BS and need to remember the memory gqs summarization many more then integration functions as a Smart Storage mechanism Smart Storage mechanism why this Smart Storage I'm saying because in the projects also we will create this okay designed to bring multiple to tools uh together and chain operations right again it is python based it is python based it is Lama index. TX in case of front end here you have Lang chain doj focused on search Centric application now this is the most important thing a broad range of application here whenever we talk about search Centric application this is the thing right many people will be talking about rag system RG right you know I hope everybody knows about rag system if you have an external customer data custom data how can you probably explore that data if you have multiple PDFs document how can you ask any question to that and probably retrieve the data itself with the help of Lang chin we can do it but if you using Lama index we'll still get more efficient results and I'm just not saying because it'll just provide functionalities to load all the PDF convert uh index those PDFs take out all the text Data index in such a way that whenever you uh ask for any query you'll be able to get the retrieval very much quicker and then for deployment it is idal for both of them it is the deployment part see at the end of the day our main aim is always to create an application using llama index Plus langin as I told you guys if I probably consider this okay now let me show you an architecture here so here is what if we are using llama index plus Lang chain now this is what is the architecture that I'm talking about so here you can see let's go from here this is your data right structured unstructured and programmatic all this data is basically indexed right this indexing basically happens with the help of Lama Index right once this indexing is happened right this this entire pipeline this entire pipeline is implemented using llama Index right it is very much efficient when compared to using Lang chin then you will also be seeing this part this entire application is built by Lang chain framework now llm models can be anything different different llm models now let's understand this so I have my structured unstructured programmatic data this indexing will basically happen with the help of Lama index now user whenever they ask any query now since we are indexing with the help of L andex usually this query when we are asking we will be quickly able to get the response now let's say we are getting some response over here we will use this response along with one specific prompt along with all the response that we have specifically getting and then we'll send it to our llm model and from this llm model we will get one another response so this entire thing can be implemented with the help of Lang chain because here the dependency will be with respect to multiple agents right multi-chain agents there may be one chain to other chain uh there may be a requirement right uh so all these things will specifically be there let's say if I want to use an application create an application which is called text to SQL right text to SQL now not text to SQL let's say I'll not use text SQL let's say I have a PDF document I have multiple PDFs document right so PDF I will Index this data whenever I ask for any query I will be able to get a response now along with this response what I can do I can create another prompt and give it to my llm model and make this llm model perform any function functionality based on the response that I'm getting over here right and then finally I'll get the response so if I'm specifically using Lang chain and Lama index I will be able to create some amazing application but at the end of the day if someone even asks you in the interview where exactly is Lama index used you should specifically say in this pipeline this is the pipeline where it will be used still here right and this squaring usually happens very much faster because the indexing technique that is used in Lama index is completely different with respect to Lin the way that it is probably creating vectors Vector Mings the way that it is creating metadata it is completely different when compared to Lang Lang chin and this is a kind of expert in this matter Lang chain overall it can do multiple things right so I hope you like this particular video I hope you able to understand this now in the upcoming video we will try to create a projects using L index and Lang chin we'll try I'll try to show you I will talk about multiple PDFs we'll talk about multiple PDF how you can query it we'll talk about amazing use cases that we are going to discuss we'll talk about what all functionalities you have in L index you we'll also be talking about Vector embeddings in Lama index uh by using this Lama index many things will be probably coming and what all applications you'll be able to develop right so yes this was it for my side I think I hope you like this particular video I'll see you all in the next video have a great day thank you one all take care bye-bye

Channel: Krish Naik

Views: 26,258

Keywords: yt:cc=on, Llamaindex tutorials, langchain tutorials, open ai tutorials, langchain vs llamaindex, krish naik generative ai tturoials, llama index tutroials, VectorStoreIndex, SimpleDirectoryReader, VectorIndexRetriever, lama_index.response, llama_index

Id: 1eym7BTnuNg

Length: 15min 22sec (922 seconds)

Published: Mon Jan 29 2024