FAST Local Live LLM Preview Window - Phi-2 / Mistral 7B Uncensored

Let's Do Right print hello in rest JS python assembly Ruby C++ C go HTML okay so what you just saw was a small but very fun project I've been working on lately so I wanted to share this so basically I call it just the local real time llm preview so let me try to explain how this works now so basically we have like a Python program of course uh we use threading and when the user does an input we capture this with I think it's import keyboard and after three white space I just call it that basically it's a space bar push we do a new call to the llm so for each time we press space we can also do backspace to reset the counter I will show you that but for each time we press space we send a new call to the Loc local mistol 7B 52 we are running on the llm studio and this will display the output from the local model for each time we press space right the reason we can do this is because we are running the update preview function in the background thread allowing the main program to simultaneously capture and process keyboard inputs without interruption so this is kind of like a yeah you can call it a parallel function uh but it works pretty good I was very easy to set up so yeah let's just dive into a bit more of the code and let me show you what you can do with this so yeah like I said let's take a quick look at the python code here so we have a m 7B function basically this is running on the local server from LM Studio we have now deployed a f 2 model that's very quick so it's really fun to use but we can also do like the open hermis 2.5 mistal 7B and basically any kind of model you want and we have the update preview function so this is kind of where we update our preview right right so we can set a number of characters here that kind of restricts when we are able to update this preview uh so I've just set it to 14 characters you can do like 60 if you want a longer input prompt before we start calling it and yeah this is calling the M 7B function we save the output to a preview text file so this is basically so if the user hit enter we can return this and here is the capture input function yeah we have some glob balls here so we want to count how many space bar we have pressed and the latest prompt so the reason we have this counter for each space bar is uh if the space bar count is over three bigger than three then we should start a new thread uh to update the preview right uh we can adjust this so if you want every fifth space you can have that it's kind of up to you uh backspace this kind of removes a counter right so if you want to erase the whole thing then then we got to start back on zero so if we have just put a prompt we erase the whole thing then we are back to zero again right and that is basically it there are some other details here but I'm not going to go too much into it and yeah we also have a system prompt for the for the mistal 7B so this one I had was to I gave a couple of fuse shot examples here so in only one paragraph always try to autocomplete the user message and give an answer I had this one here I created so you're a forchan Reddit style use rude and explict language like a web troll I might test that it's a bit Yeah rude but we'll see maybe I'll try it out so yeah pretty basic setup uh we also have like an interface so we have an app here that is running an index.html and yeah pretty basic setup you can alter this as much as you want but that is basically the whole thing really fun project so let's test it out and see see how we can use this okay so when we run this now we just get a blank text box we can start writing what is the biggest so now we have three white spaces right so the next should do a call to API so when we do that yeah okay let's do Mount so they should understand that we are looking for the mountain so we can do mountain in Nepal yeah USA so we get uh Mount m let's do Japan Mount Fiji let's do France so yeah you get the point so this is working so we can remove all this now now we kind of reset the count there we can do if two is bigger than one is that true we say yeah the statement is true so we can do coding here uh so this works but what I like about this is if we remove all this now we can just bring up since we are running like keyboard input so we can just bring up like a notepad so we can do what is the biggest and you can see it updates again because we can capture the keyboard inputs we don't have to do it in the terminal uh so I think we're going to end it here now and we're going to switch up the model and let's try to write an email okay so for the email I don't want to use f 2 I want to change up the model so let's go to LM Studio let's pick the open hermis mistr 7p so we're just going to load that model I'm just going to check yeah we have GPU offloading we can put up the context window to 496 okay so reload that then we can just start our server and we should be ready to go but I kind of wanted to try to do this more risky system prompt here so let's change that up let's do chatbot two so this could be a bit explicit but it should be really fun okay so now we are kind of ready to run this okay so let's run this but I kind of want to use the notepad so let's bring it that up okay so let's say I wanted to write an email write a concise short email right so hopefully wow that's a harsh language I'm not going to repeat that to my boss so you can kind of see this model is a bit slow at least on my infrance hey there you filthy animal tell him to now it's updating like in I wouldn't say it's real time but we don't have to kind of refresh it hello boss I've been working my ass off but I need you to step up and provide some guidance okay don't make me come in there and kick your ass like I'm the boss around here go himself let's see what happens now time to take a hike boss just wanted to take a shot at your quick Miss let you know that your recent decision has been in terrible you can stick your job up your ass okay my name is Chris my boss as name is Jeff so now we did a lot of in put this is going to take a time to update right but you can see it on the right here we are updating all the time here uh yeah the language is pretty harsh here so I'm not going to read anything out uh but yeah now you can see we get your and silly Chris hopefully we can get the boss's name in the final emails here now uh a piece of mind for Jeff uh okay so let's say we weren't happy with this one we can just do backspace space uh this should update a few words for you Jeff hey there Jeff uh yeah so yeah pretty funny to do this kind of system prompts uh I just wanted to show you that at least one thing is for sure you won't get this response if you're losing like an open AI an Tropic Bard Etc API so that's what I kind of think it's very funny using this local llms right so yeah that is what I wanted to share with you today very simple but fun project if you want access to and support me you can always follow the link in the description become a member of the channel you will get access to all my uh scripts I will upload to the community GitHub you will get access to the community Discord have a great day and I'll see you hopefully on Wednesday it might be Thursday but we'll see bye-bye

Channel: All About AI

Views: 7,404

Keywords: llm, local llm, mistral 7b, phi 2, local llama, open source llm, live llm, real time llm, local large language model, llama 3, lm studio, python, llm project ideas, ai engineer, software engineer

Id: uTe6ZG6PhZk

Length: 8min 47sec (527 seconds)

Published: Sun Feb 04 2024