I wish every AI Engineer could watch this.

five levels of llm apps consider this to be a framework and help you decide where you can use llm there are lot of different myths around what llms can do what llms cannot do where do you use llms today so I decided to put together this material uh in which I'm going to take you through kind of like a mental framework based on the extension or the depth in which you go towards an LM you can decide where you can fit this llm so we're going to first see what are those different levels of llms that I have put together then we are going to see slight extension of that got two different documents to take you through that so this will give you an idea about how LM is being used today and how you can use llms for your own applications to start with imagine this pyramid structure this is a very simple pyramid structure and as you can imagine with any pyramid structure the top of the pyramid or the peak of the pyramid is our aspirational goal and what you see at the bottom is the easiest that we can do and as with everything else you have to slowly climb to the top of the pyramid so you can probably hit the aspirational goal so to start with where do we use llms first Q and A a question and answering engine what do I mean by that it is quite simple for us to understand so question and answering engine is a system where you have an llm and all you are going to ask the llm is a question so you send a prompt and the llm takes the prompt and gives you an answer that is it that is the entire transaction that you have between an llm send a prompt get send it to the llm get an answer llm large language models are nothing but sophisticated next word prediction engines and they have been fine-tuned with something called instruction so the instruction fine tune models that means they can take a human instruction and get you an answer back for example if I ask a question for this what is the capital of India then the llm would process this and then llm has information about how to answer it and then it will give me the answer back the capital of India is New Delhi that's all what you're going to do with this thing so first level question and answering now you might wonder at this point that where can you use question and answering as an llm engine this is the first thing that people built like when llm started even back in the day gp22 level people started building simply Q&A bots so all you want to do is ask a question give an answer could be a homework could be a general knowledge question could be something about the world could be about science could be about anything ask a question get an answer as simple as that it's a very three-step process ask a question or send a prompt take the llm to process it give me the answer back very simple application now what you're going to do is you're going to add something to that application and that is how you actually build a conversational chat bot and to understand this better I would like to take you to my second document which will give you probably better idea whenever we are talking about llm there's one important thing that we need to understand is we have crossed the stage where llm is simply a large language model we have more than that so for you to understand that I have five dimensions a prompt a short-term memory an external knowledge tools and extended tools if you think of this as your horizontal these are your verticals these are different dimensions that can add to an LM so you have a prompt you have a short-term memory you have a long-term memory or external data you have tools and you have got extended tools so let me give you an example for each of this so that you can understand this better a prompt is what is the capital of New Delhi that's all the prompts you simply go give what is the capital of New Delhi and the llm understands it and gives you a back understanding just gives it back now shortterm memory is when you have conversational history or something in the llm that is what we call as ICL in context learning so whatever you stuff inside the context window the llm can take it that is your shortterm memory so you give a few short examples you give an example like for example what is the capital of us uh I guess it's Washington DC Washington DC and you give a bunch of examples like this so the llm knows what is that it has to answer this is a short-term memory next you have external data now you take data from Wikipedia and you keep it and then give it to the LM that is your long-term memory because short-term memory just like a computer the ram it gets reset every time you reset the conversation or the session and then tools you let llm use tools like calculator internet python terminal and all these things and extended tools is when you expand this much Beyond it I hope now you have understanding about the five different dimensions that we have in llms a prompt a shortterm memory or in context memory a long-term memory or external knowledge external data or custom knowledge tools like calculators and python Ripple and extended tools that goes much beyond that what we do not have currently so these are different dimensions now coming into what we wanted to see is chatbot so how do you make a Q&A bot as a chat bot is very simple now at this point you might have already got this idea so you take a prompt and you give it to the llm where you can have shortterm memory me in context memory in context learning for example so what is the capital of India so you what is the capital of India you ask and the llm answers New Delhi this is what happens in a simple q and a bot but how do you make it a conversational bot or a chat bot by adding a new dimension called shortterm memory and how do you do that you keep all these things that you are conversing into the chat conversational history so what this gives the ability for an llm to do is when you say what is the capital of India it says new D then you can just simply go and say what are some famous Cuisines uh there so at this point the llm would have an understanding you're talking about New Delhi because that conversation is stored there in the lm's shortterm memory or the in context memory so the llm can do something called I in context learning and give you the response back and that is how you upgrade in the pyramid by building a Q&A Bard giving a new dimension call history and then making the Q&A bot a chat bot so that it can converse now chat bot has applications everywhere that you can turn towards youve got chatbot in customer support you have got chatbot on websites you have got chatbot for Education like you've seen a lot of demos from Khan Academy so chatbot is quite versatile it almost has its purpose in every single business or domain that you can think of now people were using chatbot um but you know chatbot itself is not enough why we already know the answer to the question can you pause and answer if you know the answer so why is that chatbot is not enough uh for a lot of use cases the answer to the question is chatbot stops with only short-term memory you need long-term memory or you need external memory see for example I ask what is the capital of India it says new what are the famous quins there it will give me an answer quite valid llm is doing its job so let's say I'm a I'm a company okay so I'm I'm an organization let's take uh Apple for an example okay now I ask what who is the CEO of Apple of course the internet has information about it so it will say Tim Cook that's quite easy now if I go say who is the manager of the team handling iPhone 16 will it answer no I mean it might answer because it hallucinates a lot but the answer would not be correct and that has become a big bottleneck in a lot of Enterprise use cases because you do not just need internet knowledge you do not just need the knowledge that the llm has got you need more than that and that is the custom knowledge component or the external knowledge component that you need the dimension that you need to make your llm slightly more than just a chatbot and that is where a new technique called rag comes into picture retrieval augmented generation where you use the knowledge that you provide or you call it a long-term memory you use the documents the internet the sources everything that you have around and you use that knowledge to send to route to llm and then make the llm use the leverage that knowledge and now at this point probably you might have guessed it see first we had only prompt one dimension second we had shortterm memory two Dimension now we have external knowledge which is three dimension so this llm is at the center of three different things you have got prompt you have got um short-term memory and you have got long-term memory to make you understand this better uh so I'm going to take you to the rag so how does a rag look like so you have got the llm at the center of it you have got your data somewhere available so it could be on different structures it could be on database most organizations have data in their database structure database rdbms database then you have got documents which are unstructured like PDF HTML files internal portals blah blah blah blah blah then you have got apas let's say you are a sales team uh probably your data is in some CRM or Salesforce right so you need a programmatic call to make the call and get the answer back so your data could be of these different places could be like structured database like rdbms system it could be unstructured documents uh PDFs uh HTML documents anything that you have locally and then you have got programmatic access like you're a marketing team you need data from Google ads you a sales team you need data from Salesforce you are your company is heavily into it so you need data from AWS like billing cost and all other things so this is programmatic so you use one of these methods a structured passing or unstructured passing a programmatic call and take all the input data and create an index an index is what Google creates at every single moment you have got all these websites what Google does is Google creates this index so it is easy easier for Google to go Travers when somebody's asking a question and that's how Google became popular before Google people were using totally different thing Google came up with something called page rank algorithm at the fundamental of page rank algorithm you have got this index with the different parameters of course and definitely we're not building Google but so index is what we are building it makes it easier for you to understand what is inside the data so now a user comes in asks a question what is a question who is the manager of iPhone 16 team so so that question goes to the index the in this this system particular system takes that and picks only the relevant information see this index might have information about all the teams iPhone 16 Apple Vision Pro billing accounting procurement marketing blah blah blah blah blah so it has all the information what you are interested in is only this particular piece which is what you asked which is iPhone 16 manager so it this particular part is where it takes only the relevant information from the index and then it matches with the query uh The Prompt that you give and then it finally gives you sends it to the llm The Prompt what you asked and the data that you extracted and it goes to the llm llm gives the answer back to the user this is quite different from the chatbot application if you see I'll give you an example why so in the chat bot all you are doing is you have a memory question is there sometimes you might do uh let's say a long-term memory by doing user profiling I'll I'll ignore this for now you don't have to use this now ignore this for now so what you're doing is you have a question you're sending it as a prompt and you have memory that also goes to the prompt because that's how you can do it and you have llm answering this question and you get the answer back now you might ask me hey why do I need to put my thing in the external data and create an index rather why can't I keep it in memory if you have got this question at this point that is a very important question and you are thinking in the right direction in fact people who reached at this point you can tell me whether you know the answer or not the reason why we cannot do this uh or we could not have done it early in these days of alms is due to an important factor called CTX window what is CTX window CTX window is nothing but called context window this internal memory and question or the short-term memory and the question is bounded by what is the context window of this particular l so you have an llm the llm might have context window like 4K which is quite popular these days or 8K and even G like LMS have like 1 million as context window so context window is there now what you are actually doing here is you have a question the llm answers so you have a question one right and answer one comes back then you have a question two then you have answer two by the time you go to question three what you are sending to the llm is not just your question 3 you are actually sending all these things right so let's say this is 2K this is 1K answer then again 2K question 1K answer and let's say this is a 2K question so at the end of the day when you are hitting the third level of conversation I'm kind of exaggerating but let's say 2 + 3 uh 2 + 1 3 3 6 8 so you already hit 8K so conversation context window so if you have got 8K token model at this point your model will hit out of memory error or it cannot hold it in shortterm memory and that is exactly why you need rag ritual augmented generation because this one is not bound by the conversation of course you are going to keep it in conversation but you don't have to stuff everything inside your question rather you can keep it inside your index right because you already indexed and you can keep it and only the bit that is relevant comes to you and now you might be asking how is that possible and for that you know you go into like a separate tangential side that talks about semantics and uh semantic search and all the other things embedding semantic search that is quite out of scope uh if you want to go deep you should read rag llama index is an excellent library for you to read about rag uh they have got really good developer relation system uh they have got a lot of Articles uh and you should definitely read about llama index and rag if you want Advanced rag but I hope you get the point going back to our system that we put together so what do we have we have a Q and A system at the front which just takes an input gives an output nothing else then you have got the chatbot the input plus history goes together that is always short-term memory you get the output the output also goes back to the input that's why you keep the conversation history then you have got a rag retrieval augmented generation the reason why it is called retrial augmented generation is because you have got a retrieval component that you augment with the llm component and then you generate the response back so that is retrial augmented generation and the applications are enormous there are a lot of startups in in 2024 when we are recording this lot of startups just doing rag so if you can build a rag solution today in 2024 you can probably even raise F or you can be a good successful SAS there are a lot of companies making really good money solid money out of it I'll give you an example in fact like one thing that I've seen site gp. if you go to site gp. it says make eii your customer export Export customer support agent and I know this is this is a product that is making a lot of money um hundreds and thousands of dollars and at the foundation of it it is a rag it takes all the information that is available in your website indexes it or we call it data injection injection and index is set and when you ask a question it just gives you an answer back that's it it's not just a normal chatbot it is a chatbot that can answer based on the existing data so if you are breaking into llm today I would strongly encourage you to do some rag system that is by default something that you should do so if you're University student watching this if you're an early in career professional I would say you should build a couple of rag examples so you know there are a lot of no aners in rag like how do you improve indexing how do you improve indexing by changing chunking what kind of algorithms you use for embedding and what kind of models are good with rag whether you put the text at the top is it good whether you put the text at the bottom is it good good if the text is in the middle it is good a lot of components to rag rag is not just simply what we discuss usually on this channel you can go Advanced Rag and I would strongly encourage you to spend some time in drag unless you want to get into something that is quite exciting and interesting but before we do that I would like to quickly show you one more thing that not a lot of people discuss when we talk about llms it is not necessarily rag it is just like using short-term memory so it doesn't use long-term memory but it has its own potential which is to use llms large language models for classical NLP task classical NLP Downstream tasks for example let's say you want to build a text classification system what is a text classification system you give a sentence for example uh the movie was complete crap now is it positive or negative positive or negative you choose you build you train a text class classification model just to figure out this for example or the other example I can give is you have a review let's say the movie was amazing and the actress um was exceptional now you try to build a model that will say what kind of review is this for example is this review about movie um theater or director or actor so now you know this is an actor based so this is what Tex classification in classical nlps there are lot of other tasks that you do in classical NLP what you can do is without having to build your custom model like let's say bird based model XG boost based models you can use llms large language models for classical NLP problems because large language models have really good in context learning and with the current memory that you have got with a few short examples or tree tree of thoughts or a chain of thoughts you can make your large language models a good zero NLP classifier or it is applicable for lot of other tasks as well so one thing that not a lot of people are exploring I would encourage you to explore if you work in classical NLP problems like labeling or a text classification entity recognition whatever it is you can leverage llm now the question is do you want an llm based Solution that's a different topic I'm not talking about you know you looking for a nail because you have got a hammer I'm just saying that this is a good opportunity wherever you don't want to build models you can use this but of course if you can build models that will be probably cheaper than you know making calls to llms and getting answer back but summarization text classification entity recognition I think llms are exceptional zero short llm uh and down for Downstream tasks and you should definitely leverage it now now with this we have arrived at rag okay so we have arrived at Rag and we already know what is rag now we are entering into a very interesting phase about what everybody is obsessed with what everybody's love agents very recent announcements from Google Microsoft previously open Ai and every announcement you would have seen two important things as a common Trend one is you would have seen multimodality multimodality what does it mean it just simply means instead of just chatting with text you can chat with images you can ask questions in voice it can respond back in speech you can send videos so one important Trend that you are seeing is multimodality and the second important Trend that you see everywhere is Agents multi-agent setup where you have got multiple agents you can summon these agents to do certain tasks and these agents will do it for you just like men in black MIB they have a purpose and they will do certain tasks but before I jump into agents I want to actually introduce you to another important concept called function calling function calling is the precursor to llm agents in function calling what you do is you have a short-term prompt you you have prompt you have short-term memory uh sometimes you need external memory sometimes you don't need external memory but you are giving the ability of calling external tools and you are giving the ability of calling external tools by doing something called function calling function calling to be honest is a terrible name cuz you're not calling any function here you're you're not making the llm call anything not at all all you are doing is you're forcing the llm to give a structured response back so you can call and I'll give you an example what is function calling so let's say that you have a weather API okay weather I think everybody goes with weather AP so I'm going to I'm going to skip uh let's say you have a currency converter okay currency converter what kind of things a currency converter need okay you need input currency you need output currency you need date you need amount technically these are the four things you need what is the amount that you want to convert for what is the input currency what is the output currency and what is the date for which you want to do currency conversion let's keep as a simple APA now typically when you go to an llm okay and say what is USD to irr today first of all llm may not understand what is today llm might know USD llm might know INR but lm's memory is frozen because it's a snapshot see a large language model is a snapshot so it memory has been frozen to let's say September 2023 or something like that okay so what it cannot do is it cannot give you the latest information and you cannot do this with I mean you can do this with rag kind of like you can every day take knowledge ingest keep it in your memory and then give it back not very efficient um expand this to stock market a daily data doesn't matter because everything changes like every minute and every second so you need something instant what you do you call an API if you are a programmer that's what you would naturally do you call an API now if you want to call an API uh what you need to call an API so let's say this is the information what I need at the end of the day I want to call it currency converter right and I'll say input output date amount right I need to make a call like this so I need four arguments that is a solid input could not be like oh United States dollar and some other time I'll be like USD some other time I'll be like US dollar I mean that will not work right you need a specific format for everything your let's say amount should be an integer right a this should be a date object so you need to force this llm to give you a particular response back otherwise what happens is this LM will throw you anything for example what I want to say is what is USD and I so it'll be like oh USD and I Soo on September 2023 so you have to force guide the llm to make a particular type of output and somehow universally everybody has accepted that format is going to be Json EX except anthropic which absolutely loves XML so if you use anthropic you use XML if you use any other model you use Json so you're forcing an llm to give you a structured response back a Json that can help you make this function call you can call this function with that Json so a guided response into a Json is what everybody calls function calling you don't necessarily call the function and function calling but you get the output that will help you call function call right clear now that is exactly what is a precursor to agent because in a function call you have the ability to call a function and agents are nothing but a bunch of function calls stitched with tools so what do we have in agents we have a bunch of function calls plus tools and I would like to introduce to you a very interesting solution that can help you understand more about a agents if you are too old in the AI world you would have probably recognized this immediately and this was the workflow of something called Baby AGI so baby AGI was quite a popular thing back in the day I mean back in the days like less than one year before I guess or maybe more than one year a function call is what I said is the foundation of Agents but what is an agent now if you have seen our pyramid you would know know our agent sits right at the top like closer to what we our aspirational figure is now what is this agent how do you define an agent so it's simple first of all a chatbot and a rag all of these guys if you see here they end a text or you know some kind of thing like input output images video all these things right that's where they in one of these modalities they're done what you achieve with agent is something that is absolutely stunning you don't stop at text response you stop at an action you trigger an action and that is what agents are simply you take llm you connect them with tool you give them a purpose or goal that is your agent and that is exactly what baby AG has done back in the day like there are multiple agents now but if you see baby a which is a very wonderful framework you can see that there is a task like there is something that has to happen there are certain tools like for example Vector DB and all the other things are there and every agent has a purpose like okay you have to execute you have to return you have to do something you have to do something and they have a goal so you have tools purpose SL goals and llms and this all together work for a common goal and that is your agent there are multiple agent Frameworks that are quite popular these days is crew AI L graph you have got a py autogen and most of these things you will see first you have to define a role you have to refine a goal Define a goal a role goal and then you have to save which llm that you want to use as a backend engine and then you put together a system of one this is single agent now you put together like this is a team that is your multi-agent setup with agents people are doing amazing things you can make make an agent book your ticket you can make an agent let's say read something um distill something create a note publish the blog post you can summon these agents to do a lot of things and personally for me uh the most time that I spent reading about agents because you it's it's becoming quite obvious that agents are the next Frontier in uh the way we can take llms forward I mean there are a lot of different things but at least personally I'm quite interested in automation usually and I think agents are going to be the next big thing in I mean currently itself is a big thing Google has got Google's own projects like they call their own agents I don't know what they call they have a lot of different names opena has its own agents and uh every time you talk to some company you speak about agents because you want to summon these agents you want to connect these llms to like different dimension and on this Dimension that what we are connecting is the tools Dimension so you take llms you have the function calling ability and once you connect them to to tools you are unlocking the potential of something immense and that is what you call as agents I'm not going deep into agents because this is probably I'm hoping it to be a series depending upon how you all like it but in the series my next focus is going to be agents so agent is quite closer to the top and that takes us to the almost the end of the video which is what is our aspirational thing what is that we are all trying to go towards to which is L LM OS and this is inspired by Andre kPa who created this amazing structure so what is happening here this talks about using llm at the center of a conversation or sorry center of an operating system if you go back in the day computer was created just for simple calculation purpose right you want to add a and you want to add a and b you want to keep a for one and B for two and then you want to add them that's that's what like initially computer was started like very very very back back in the days then computation started increasing computation started becoming less expensive more compute then we have the computer that we have today and garpa is arguing can we have a similar vision for llm and where the vision is you keep llm at the center right you keep llm at the center and at the center with llm you have Ram which is the shortterm memory or the context window then you have long-term memory the diss system that can be used with rag then you have the agent structure that you have with tools and then you connect it with internet and when you connect it with other llms to have like a multi-agent setup or like a peripheral setup and then you have your peripheral devices where you have got audio and video can we put together a system with all these things working towards a common goal and that will ideally become your large language model operating system this is quite a vision at this point there are certain implementations available at this point those implementations are based on current understanding they are mostly let's say llms plus function calling plus agents multi-agent more tools that is what the current llm OES it's not like a radically has a different total View altoe and that's why if you see even in my framework that I've created llm o is currently developing and it is everything that we have got the tools the extended tools the peripheral tools with long-term memory with shortterm memory just one input from the user where it can run itself and then it can execute certain things I think that is a future that we are heading I'm not sure when we are going to do it but uh if somebody says something a for me today a could be like this could be like the baby a I mean I don't I don't I don't trust a as a concept anytime soon but um yeah leaving the conscious thing Consciousness and all the other things out I would say llm o is at the top where we can expect something closer to a happen and all these things lead us up to there so I wanted to keep this video brief but uh this video is already going to be like more than half an hour I wanted this to be like a crash course where you understand if you don't know anything about llm OS uh maybe you have not taken any course so this is going to help you to see how the future of llm O is coming and what led us up to there and uh let me know in the comment section if you like this kind of content I'll put together more this took me a lot of time to create the framework design put it um in a particular thought process to you know make it make it understandable and this is basically what a lot of llm courses offer so I'm I'm definitely looking forward to hear more feedback and if you like this kind of format subscribe to the channel see you in another video Happy prompting

Channel: 1littlecoder

Views: 64,155

Keywords: ai, machine learning, artificial intelligence

Id: F5nlMBVZxb4

Length: 33min 49sec (2029 seconds)

Published: Wed May 22 2024