Stream LLMs with LangChain + Streamlit | Tutorial

good morning everyone how is it going today welcome back to the channel and welcome to this new video in which I'm going to be showing you how to build a chat that does pretty much what you see on the screen it streams the generation as it is being generated this is great for user experience um just to be clear I am going to be building the entire chatut here for the newcomer so you see how this is all working but if you already know how to build your chat but in streamlet and Lang chain you can just jump to the section where I show you how to use the streaming part okay so without any further Ado let's get right into the [Music] video okay so just a quick note right here uh in the description of the video you will find a link to this article that you see right here which is pretty much an article that accompanies the video here you will find all the written explanations and you will also find the quick links to both the GitHub repository and I will add a link right here also to the Google collab notebook so that you can check that out as well and everything is centralized in this article right here okay so let's take a quick look at what we're actually going to be building here first of all we're going to take a look at how the streaming generation works we will make a chatbot that we will stream the response from the language model as it is being generated we're going to use Lang chain to interact with the language model we're going to use streamlit to create the front end of the application and we're going to have the chatbot remember the chat history and show it to the user okay so let's create all of that great so let me just tour you around of this very quick uh project that we're going to be setting up okay so as as you can see I have already set up the project we have myv file with my open AI key we have my G ignore and we have my app.py inside of here of my source directory okay and also let me show you what's going on here I have already enabled my environment called streaming bot I am using cond environments if you're interested you you can just create it doing cond create create and then you just call then you just call name your environment and then you specify the python version that you want okay my case I'm not going to do it because I have already done it let you just activate it activated by calling the name of your of your environment okay so as I showed you before we have already installed Lang chain and Lang chain opening I but I also did pep installed stream lid and python. EnV to be able to use my secrets which are right here okay um if you want to see more details into how to initialize or set up your project I'll link a video up here so that you can take a look at that but for now let's actually focus on building the application okay so the first thing that we're going to do is we're going to set up the user interface for the chat but okay in this case remember that in order to run our application in streamlit we can do streamlit run and then the path to our application so there you go it started off this application that you see right here and as you can see it only has this title because that's the only thing that I have added right here so what we're going to do is right now we're going to add the part where the user adds an input okay so let's actually create the entire chat interface let's do that right now so I moved myself up here so that you can see both the chat but up right here as it's being built in real time while we code up here okay so what we're going to do right now is we're going to initialize the entire chat interface now in order to do that remember that uh stream lid actually comes with a very convenient uh chat uh module that you can use so in this case we're going to use chat input and here it takes whatever placeholder you want so I'm going to say your message there you go and let's say that this is the user query like that if I save and up here I mean you don't see it because there's my face but there is an always run rerun and now we're seeing the changes life so now once that we have this what I want you to do is we want to initialize our chat history variable okay and that's what we're going to initialize let's initialize it up here so let's say if chat history not in SD session State because the SD session state is the um the object that is going to keep all of our variables all of our variables persist throughout the stream L session we're going to put it right here and if it's not being initialized we're just going to initialize it as an empty array like this okay and then from the Mt array we're going to actually log our messages but now that we have this user query and the chat input what we're going to do is we're going to deal handle whenever a user enters a user query so let's say that if user query is not none and and if it is not an empty string what we're going to do is we're going to first of all take the chat history and we're going to append we're going to append the user query however we're not going to append it just like that we're going to use l chains schema for a message okay so from L chain core. messages we're going to import human message and oops and we're also going to import AI message and since this one right here is a human message we're going to import it as a human message like that and after that we're going to get um we're going to show it in the in the application so let's do with st do chat m message and here you say who is actually talking in our case this one can be a human so if you write human and let's just type here the mark down and let's say that it is the user query okay so let me just show you what this is actually doing if I type right here hello and I hit enter you will see the message from the human up here okay and then after we do that we're going to want to actually create a generation for the uh chat message okay so let's do for St do chat message and we're going to say that this is the AI we're going to say that the AI response is going to be I don't know and right here we're going to just add the markdown for the AI response and there we go so hello hello how are you seems to be working correctly okay but so far it's only showing one one part of the message what we're going to do right now is well first we're going to append that AI message to our session stage just like we did with the human but now with an AI message schema and now let's actually just show the conversation right here [Music] conversation there you go so the conversation we're going to say that for message in s session stage CH history If the message is an instance of human message we're going to do chat message human and we're just going to show the content of the message and we're going to say else I mean if it's not a human that means it's the AI so we're going to show the hum the AI message like that seems to be working correctly and we seem to be appending the thing correctly um [Music] else there we go so let's see how this looks hola I don't know how are you I don't know perfect now the conversation seems to be running smoothly and the user interface seems to be running smoothly as well okay now let's actually just create the get response function which is actually going to generate the response using um a language model okay and we're going to create a very quick chain using langin for this so stick with me for that all right so let's just add some comments right here here is the user input and here let's let's just create the get response from the chat from the language model okay so we're going to define a function and we're going to call it get response and this one is going to take the query and it is also going to take the chat history okay now the first thing that we want is we want to create a template for our for our prompt because we're going to be using a prompt here and I have my template prepared up right here let's say you are a helpful assistant answer the following questions considering the history of the conversation we're going to we're going to pass in the history as say variable right here and we're going to pass in the user question as another variable right here okay now second thing we're going to do is we're going to initialize our prompt from this template um however for to do this we're going to have to import another thing right here let's say we're going to import from lank chain core because this is a core module we're going to import from the prompts module we're going to import chat prom template okay so that's what we're going to be importing here chat prom template and let's say from template and we're just going to pass in the template that we just created okay now that promt um the second thing we're going to need to for our chain is we're going to need a language model remember we're going to be using open a just to keep things simple so from longchain open AI we're going to import chat open AI like that so our llm is going to equal chat open AI like that and last but not least we're going to add a string output parser and that also comes within Lang chain core so from Lang chain core dot output parsers we're going to import a string output parser there you go so now we can actually create our actually this is not correctly indented there you go now we can actually create our chain that is going to I mean remember that we're going to be using L cell so remember that the chain is first of all we put the prompt prompt then we're going to add the language model and then we're going to add our string output parser right here and now we can actually use the chain. invoke as we said before and this one right here takes an object with the variables that we have right here okay in this case remember that our variables as as you can see up here is chat history and actually this is supposed to be like this and this one is going to be the chat history that you see right that we are passing to our function like that and the user question is going to be the query that we're passing right here and let's just return whatever this is okay so save and we can actually instead of saying I don't know every single time we can say get response or we can pass the user query first of all and second of all we're going to pass in the St session State chat history because remember that the second parameter right here has to be our our chat history and that chat history is stored in this session state that we have right here so let's save this let's rerun this and let's see how this works hello how are you now it as you can see it's actually thinking before it sends the the response so that's not what we want but let's just continue testing it I am Alejandro and let's see if its memory is working what is my name there you go my name is Alejandra so the memory is working the chat is working everything seems to be working now it is time to actually create the streaming feature so let's do that right now okay so now let's actually use the this new feature from streamlit that allows you to display whatever a generator is returning okay so let's go back to our get response function right here and remember that so far we're returning a chain do invoke uh whatever the chain. invoke method returns and as you remember from The Notebook this one Returns the actual completion what we want to do is return just the generator okay so in order to do that remember that since chain is a rable we can do chain dot stream or we could also do a stream if we were using a synchronous functions and in this case this is going to return a generator okay so here if instead of doing this we if if we do if we leave it like this let me show you how it what it actually does how are you and then as you can see it returns a generator it doesn't return the actual response and that is what we want but we actually want to yield each generated token by this generator so let's see how that works in order to do that I am actually going to put this thing right here inside the STD markdown like that and right here that here you could do what we did in the notebook do a for Loop for each part of the generation however streamlet recently introduced a very nice module that allows you to do this with a single with a single line of code so let's use their new module called write stream like this so if I save this and I hit I mean of course I need to I need to redefine my AI response like that and now if I reload this let's see tell me a story all right what is kubernetes and there you have it it is streaming the contents now as you can see let's see if the memory is still working let's do tell me a poem about that so in a b digital C kubernetes sales Etc so it seems to be working the streaming seems to be working just remember to use right stream if you're using streamlet and remember that you have to run the generator inside of this right stream for it to actually work okay so there you go that is how to use a generator and how to display the streamed contents from your language model using Lang chain and streamlit I hope that this was a useful tutorial for you I hope that you have learned a lot and I'm very happy to see you here please subscribe join the Discord Community to actually have um a closer uh relationship with the community and I will see you next time [Music] [Music] [Music]

Channel: Alejandro AO - Software & Ai

Views: 20,288

Keywords: llm, chatgpt, llm project ideas, llm project, langchain in python, langchain prompt, streaming llm, streaming llms, stream llm, streamlit llm chatbot, streamlit tutorial, langchain streaming, langchain streaming response, langchain agent streaming, langchain streamlit, langchain streamlit chatbot, streamlit chatbot, streamlit chatgpt, chatgpt clone, langchain stream, gchain stream response, langchain lcel, openai, open source llm models, local gpt streamlit, streamlit openai

Id: zKGeRWjJlTU

Length: 18min 25sec (1105 seconds)

Published: Fri Mar 01 2024