Run Any Chatbot FREE Locally on Your Computer

have you ever wanted to run a chatbot locally on your own computer you know your own version of like chat GPT or anthropic or something like that but do it completely locally like don't even need to be connected to the internet all of your data and information that you type in is completely secure and only on your local computer well I think I found the absolute best tool to do that with and it's free and open source check it out it's called Jan and you can find it over at jan. a you can see all of the code is completely available over on GitHub so you can download the code modify it build your own version of this tool if you want and under the license you can see Jan is free and open source under the agplv3 license now one thing that's really cool about this Jan app is that it works on pretty much everything you can download it on an M1 M2 M3 Mac an Intel Mac windows or even Linux now you've probably heard a lot about open- Source models like llama 7B and mistol and various other open- source models but it's always been a very confusing and sort of complex process to download and run these models on your own computer this tool seems to make it really really easy so let's go ahead and put it through its motions let's go ahead and click download for Windows here since I am on a Windows PC here I'll go ahead and run the install and just like that you can see I now have a hub where I can have chat messages but at the moment there's no models installed I don't have a llama or a mistol or anything like that installed to be able to use the chatbot we can see here it says you don't have a local model yet explore the Hub I click here we can see what models are immediately available to download and use so we've got mistol instruct 7B open Hermes neuro 7B Trinity Starling Alpha llama 2 Trinity V1 and even the newer mixol 8X 7B which is probably the best opsource chat model on on the market right now because it uses eight different 7 billion parameter expert models we've talked about this in a few previous videos and if you want it to give you the exact same types of outputs that you get out of something like chat GPT it's even got the open AI GPT 4 model that you can use you will have to plug in your own API key it's also got GPT 3.5 turbo 16k and the original GPT 3.5 turbo so not only can we use the open source model but we can use a handful of the closed Source models as well and the cool thing about this is you can set up multiple large language models and then sort of switch between them to see which one you like the best so let's go ahead and install mixol 8X 7B now this is a 26.440 this one's for 4.8 GB now it looks like the two versions of GPT 3.5 turbo and GPT 4 are already pre-installed for us so we don't have to actually do anything we just need to keep in mind that if we are going to use one of these open AI models we will need to be connected to the internet because it's going to use the API it's going to connect to their servers and it's going to do all of the work over there so you will need to be connected to the internet if you want to use one of these models but they are available and we can compare them with other models if we'd like now that I've got mixl 88x 7B installed as well as llama 27b I can come back to my chat page here and if I open my little sidebar you can see I've got some options so I can give this new chat a thread name let's just go test chat one you can see it calls it test chat one over on the left side very similar UI to chat GPT then underneath the assistant tab here we can actually give some custom instructions and under the model tab we have a drop down where you can see our two local models mixol 8X 7B and llama 27b as well as the three open AI models that are available now if I do want to use let's say for instance GPT 4 I can select it and then I do have to enter my API key here in order to use it you can find your API keys by going to platform. open.com ai- Keys create a new secret key I'm going to call it example key create the secret key and then I'm going to go ahead and delete this key when I'm done with this video so it won't continue to work but I can copy this key here jump back into our platform and paste our API key in let's go ahead and boost the max tokens all the way up here I'm actually going to jump into my chat GPT account real quick grab the custom instructions that I've given it in the past use these same custom instructions in my custom instruction box here and now I've got custom instructions set up now I can chat with it just like I would chat with chat GPT now because I'm using my API key it's going to cost me a few cents every single time I ask asking a question but this is just for demo purposes more than likely if you're using this Jan app you're going to use it for the free open source models like llama and mixl but let's give it a sample prompt of give me 10 strategies to grow my YouTube channel and click Send and you can see just like chat GPT it starts to Output a response in real time and there we go it quickly gave me 10 ideas on how to grow on YouTube give me 10 more ideas it gives me a handful more ideas if I jump over to platform. open.com usage I'm actually recording this on January 3rd but this thinks it's January 4th already and you can see I spent 2 cents using GPT 4 already this one will cost you money if you want to use GPT 3.5 or GPT 4 and now let's go ahead and create a new chat I'm going to go ahead and use my exact same custom instructions pull this into the new chat here we'll call this one test and we'll call it mixol now if I select the model mixol 8x x7b crank my Max tokens up I can use this large language model as much as I want with absolutely zero cost it's completely running off my hard drive the only cost is the power my GPU is using while running this let's give it the same prompt give me 10 ideas to grow a YouTube channel fast and using mixol 7B it starts to give me a response to real time now you can't hear it but my GPU fan is really kind of cranking up right now we can see down here we're using 59% of the CPU and 74% of memory and just like that we got 10 pretty good responses on how to grow a YouTube channel and I would say the level of output is fairly close to what you'd get from GPT 3.5 and once again let's go ahead and create a new chat we'll call this one test llama 27b I'll paste in the same custom instructions right here and then the model we can switch to llama 27b I like to crank up the tokens here here and let's try a different prompt what are some best practices to get high engagement on Twitter it's firing up the model real quick you'll see my CPU and memory usage crank up as well there it goes right there I click on this little icon down in the bottom left you can actually see how much RAM and how much CPU it's using and it starts to give me a response and the token speed here is actually much faster than what I was getting out of chat GPT and what I was getting out of mixl so pretty interesting there but again there are just a ton of models that we can play with in here they're open source they're free they're installed locally so you don't have to worry about the privacy concerns of it I could disconnect from the internet right now and this chat would still work for me assuming that I'm using llama 2 or mixol and I can download any models I want off the internet I can download uncensored models I can find models that other people have fine-tuned and set them up in this Jan app and run them locally on my computer this has to be the easiest fastest freest way of running chatbots like chat GPT locally on your computer and right now I'd say the available models are about as good as what you're getting out of the free version of chat GPT or GPT 3.5 so if you've been wanting an easy way a shortcut to quickly access and use offline local opsource chat models this might be the best entry level solution I found so far it's really simple it's free it took me less than a minute to download and set up yes there's some time to download the models but that was even fairly quick I mean even the 24 gbyte file only took 2 or 3 minutes to download and we get to use it in a very recognizable very intuitive user interface because it reminds me very much of what you get out of chat GPT just with a few extra options over here on this side this video is not sponsored I have no relationship with this Jan tool or any of its developers I just came across it today thought it was really really cool thought it was would be worth sharing because it's probably the easiest way I've come across to use these local models and wanted to quickly share it with you so hopefully you found this valuable go check it out again totally free jan. it's open source and they're continuing to build off of it and iterate off of it and as more new open- source models become available this might be the go-to easiest tool to run them on your computer that's all I got for you today if you want more quick tutorials like this the latest AI news breakdowns research and other cool Ai and Tech information give this video a thumbs up and I'll make sure more like this show up your YouTube feed also maybe consider subscribing that helps the channel a lot as well if you haven't already check out future tools. this is where I Cate all the coolest AI tools that I come across like this Jan tool that I just showed you I update it every day along with the AI news I have a free newsletter that shares just a handful of Cool Tools every week and the most important news of the week it's all totally free for you over at Future tools. check it out if you haven't already and thank you once again for tuning in really really appreciate you see you in the next video bye-bye

Channel: Matt Wolfe

Views: 110,880

Keywords: AI, Artificial Intelligence, FutureTools, Futurism, Machine Learning, Deep Learning, Future Tools, Matt Wolfe, AI News, AI Tools, open source ai, mistral, llama, chatbot, mixtral, mistral ai, mistral 7b, llama 2, artificial intelligence, ai chatbot, mixtral ai, chatgpt, ai, mixtral 8x7b, llama 2 13b, llm, llama2, chat gpt, gpt 4, open ai, openai, hugging face, ai model, gpt4

Id: zkafOIyQM8s

Length: 9min 59sec (599 seconds)

Published: Thu Jan 04 2024