Building a local ChatGPT with Chainlit, Mixtral, and Ollama

in this video we're going to build a sort of mini chat GPT that we can run locally we'll be using the mixt large language model via llama we'll interact with that via the light llm library and then the chat part is going to be provided by a library called chainlet so let's go over to the terminal and we're going to create a file called chat. P we're going to then import chainlet at the top and then we're going to introduce this annotation cl. on message and then we'll Define the function it takes in a message so what happens is every time the user writes a message it's going to come in here and so what we're going to do is we're just going to Echo the user back the message and then we'll call message. send now we're going to go over to the other Tab and we're going to run chainlet run and we'll pass it the name of our file and then we're going to pass in The Hyphen W that means it's going to watch the file and update and then minus H so that means it's going to run it in headless mode and we'll manually launch the browser let's now open the browser and so this is what we'll see when we first launch it and we can type our message in at the bottom so we'll type in hello chainlet and as you would expect it very very quickly comes back Echo hello chainlet okay so far so good now let's get it to do something more interesting so we're just going to update that message just so that it's empty so we're going to send an empty message as soon as as as soon as we receive the user's message so that it knows hey we're actually doing something and now let's go up to the top and we're going to import light llm and if we now come back down into the on message function we'll create a system message we're then going to create ourselves a variable an array of messages we'll put the system message in there and then we're going to also append into that the message that just was sent to us by the user now we're going to call our llms we're going to use the a completion function so that everything remains as synchronous we'll pass in the name of the model pass in that array of messages and then make sure that we put the respones streaming so that we can see the response as it comes back rather than Allin one go then if we come down we're going to iterate over the response we'll then pull out the content from each chunk um and then we'll add it into that message that we created at the top and then once we've done done all that we're going to update the message so let's come back to the browser again and let's see if we can ask it a question so I'm going to ask it what is Apache kafa in up to five bullet points let's see what it comes up with so I think this is a this is probably a reasonably reasonably good reasonably good answer I would say let's now ask it about topics so can you explain topics to a 5-year-old and you can see it doesn't really know what I'm talking about because it doesn't have the currently have the context of the previous message so every time we do a message it sort of starting from scratch so let's fix that so if we come back to our file and we're going to use another annotation so this is called the on chat starts we'll have the start chat function let's just go down and get our system message and we'll bring it up into here and then we're going to create a user session variable called message history and we'll put the system message in there and then if we come back down into our on message function we're going to update the messages variable so rather than just being a variable we use that session history variable instead and then finally let's come down to the bottom and make sure that the respon want also goes into that messages array so that the the chat bot knows all the history of what we've asked and what the answers have been let's save the file and then we'll come back to the browser again and what what's cool here is we can actually go and see like what were the previous questions that we asked we can select that CFA question let's ask that again so the answer is pretty much pretty much the same as what we had before and now let's click on the topics one and now this time it's much better right it knows we we're meaning Kafka topics rather than just a generic topic and so it comes up with a with an analogy of of what topics are I think it's it's not it's not too bad at the answer now I write a journal and I'd quite like to use an llm to explore some of the entries to see what was I thinking about what was I feeling at different times but I don't really want to upload my data to chat GPT or perplexity AI or or a similar tool like that and so chainlet quite helpful here because we can actually also upload files so let's update our code to handle that so we're going to just get that messages line and we'll just move that up to the top and then any files that you upload into the chain L app will be available and under message. elements we're going to check whether there are any and then we'll iterate over them we're going to put the content of the file so we're just assuming these are just text files we're going to put the content of the files into our messages array so I into the into the context and there that's available to the llm and then we're just going to write a confirmation message saying hey we've uploaded the file and then we'll we'll send that confirmation message straight away and everything else stays exactly the same so I'm not going to use one of my actual General entries but I've got I got chat GPT to write me one so let's have a look at that uh this is way more lcal than than the stuff I come up with the stuff I come up with would be would be really boring to read I expect but let's go now back to our UI and we'll drag that journal file onto the UI and then we need to give it a message so we're going to say hey I want you to summarize this journal entry as it comes back this journal entry describes the author's upcoming day it talks about a planning meeting and then it kind of explains some of the other stuff that they did has a team lunch and how they're hoping to make progress on the bug that they're working on so another question one question that I always am curious about is would you say is the main emotion being expressed so it says they're determined to make progress it also talks about a bit of nostalgia so I think it's pretty it's pretty good uh let's see if it can figure out are there any other people mentioned and so it kind of mentions that there are other people sort of mentioned in passing and then in particular it identifies Alex so I think it's done it's done a pretty good job I think this is going to be quite a useful tool for me to go and explore my previous thoughts now that you've seen mixol used in a chatbot you probably want to know how it compares to its cousin mistl 7B we've got you covered in this video here where you can see the responses the two models give to the exact same prompts

Channel: Learn Data with Mark

Views: 3,908

Keywords:

Id: MiJQ_zlnBeo

Length: 5min 38sec (338 seconds)

Published: Fri Jan 05 2024