LlamaIndex for Dummies: Choosing The Right Index For Your Use Case

in this video we're going to talk a little bit about the different llama index index types what they are when you would use them what use cases would you use one particular index type over another these are questions that I had as I was working through this so I thought I would make a video to share my findings so there are four index types here there's a list index a vector store tree index and a keyword table index and you can see in the documentation here there are some diagrams that visually show us what's going on right so with a list index we're creating an index sequentially off a series of nodes and a node here represents some information you know maybe it's a document or a piece of a document and then when it comes time to query our index the way the list index works is it goes through each node sequentially unless we filter by maybe a keyword or do an embed in base query which will only fetch the most similar you know let's say we we want to find three most similar nodes to our query we can do that with the list index now that also happens to be how the the vector store Works based off of embeddance um and so the vector store index is all about similarity search so it's good for semantic search because it's you know not the exact word but hey this might mean the same thing as this so I've used primarily I've used a vector store index for most of my use cases it gets the job done in terms of you know generative semantic search tree index if you have some hierarchical set of data information that you want to be able to Index this is where your tree index comes into play and then keyword table index extracts keywords from each node create somewhat of a map and then when you query now based off of those keywords that tells us what what nodes we need to uh extract our data from so metal a little cheat sheet here I guess you can call it and the important part is like okay so when would I use a list index over a vector store right and for cases where you want to summarize something a list index is going to be superior because let's say you've got a long super long document that you want to summarize a vector store index isn't going to allow you to do that because it's only going to when you query it's only going to fetch the most similar nodes to your query right so you can say you can still query and say hey could you summarize you know XYZ but it's only going to fetch three documents that are similar to x y z whereas the list index if you say hey summarize this thing it's going to go through all all the nodes of that document let's say and uh go through each one and then you know check the query and say hey do we need to refine our response based off of this query okay let's move on to the next node and so on and so forth until it has summarized the whole thing hopefully that's a decent example or let's say you need to store and retrieve like a chat bot history and what's important there is it needs to be sequential so a list index where you want to retrieve some kind of sequential data that's ideal tree index tree indexes like I said hierarchical so if you've got let's say a file system right and you want to navigate through some type of directory structure that has subdirectories or subsections then a tree index is ideal so you know quick example off the top of my head um you know let's say you actually have like a a directory where you know it's broken down by departments and then each department maybe has some teams and then each team has some employees and you want to find what employee does what particular thing and that case that I just made up a tree index would be superior than a vector store list um keyword table index I I was wondering for a long time like why would I use the keyword table index when there's this you know Vector store index which has the the benefit of like semantic search why would I I felt like keywords was moving backwards right but there are instances where you need to extract um exact matches right so like specific words and terms and phrases um so the K the example here I have is a legal document search engine find in documents contained in specific keywords you might not want a vector store for that because it might uh give you results that aren't um that aren't exactly relevant right might have a similar meaning but you know with the keyword table we can get like exact right so um hopefully you found this useful there is a little bit of overlap for example with the list index in being able to embed your query and just fetch um you know top K similar nodes like the vector store um and then llama index also offers the ability to compose index types over another index type so for example you know you can have a you can compose a list index over a bunch of tree indexes or you can compose a tree index over a bunch of list indexes I'm just making that up but um you know it all comes down to the use case and uh exactly what type of queries You're Expecting and what type of information you want to be able to retrieve using this

Channel: Omari Harebin

Views: 2,232

Keywords:

Id: TDo-PgmRVhQ

Length: 6min 43sec (403 seconds)

Published: Wed Jun 07 2023