JAN: This 100% LOCAL AI ASSISTANT is better than ChatGPT (works w/ RAG, Local Models, Groq & OpenAI)

hey there welcome back to AI Cod King today we'll be talking about Jan Jan is a desktop app that allows you to run open- Source models locally you can also connect your open AI or Gro API keys to chat with online models as well so basically with Jan you can connect all your AI models in one place and chat with them it is fully open source and is pretty easy to use you can get it installed in one click and you don't need to worry about using the terminal coding or configuring multiple files it's all pre-done and you can get it up and running in literally seconds if we look at their featur section the first one is about local you can run AI models like Llama Or Mistral directly on your device for enhanced privacy no need for an internet connection this is really great because if you want to chat about something sensitive then it's much better to chat about it locally for instance if you're working on a project that requires confidentiality Jon's local model feature ensures that your conversations remain private and secure but if you don't want that you can connect it to remote models like open AI Gro or Mistral API this way you can access its capabilities without needing Advanced Hardware Ware this flexibility is particularly useful when you need to tap into the capabilities of online models but still want the option to switch to local models when necessary all your conversations are also stored offline it is also crossplatform and available on Mac Linux and windows making it more accessible one thing that I also like about it is that it also exposes API endpoints for you to use these API endpoints can be used for your custom applications or other AI applications you may use also these API endpoints are open AI compatible so you can use it with anything that supports open AI models it also has the extensions option to set up some other features if you want to for example you can add custom plugins to enhance J's functionality or integrate it with other tools and services you can use it with PDFs docs and any such text file that can be parsed it has two built-in engines for inference the first one is llama CPP and the second one is tensor RT LM both are good but by default I think John uses llama CPP the Dual engine approach is better as it provides you with more flexibility and options when it comes to model inference you can also connect it with LM Studio or ama's endpoints if needed now that's enough talk let's get it installed and check it out I'll be telling you how to use it locally and with Gro models first come to Jan's website and click on the download button then you'll be asked to select your operating system choose it and get the installation file downloaded go ahead and get it installed once you have it installed you'll see this interface looks pretty cool now you can see that it says you don't have a local model yet so let's click on the explore the Hub option now here you'll see multiple options for the models you can also paste a hugging face link and it will download it automatically but I'll be installing the llama 3 Model so just hit the download button here once downloaded you'll see this use button click on it and you'll be sent back to the chat page now here you can see the Llama 3 Model is selected now let's send a Hello message it initializes the model at first and now it gets the response generated pretty cool you can rename the thread over here as well you can can also change the instructions for your assistant here as well then you can select the models over here and then you have the infer settings here you can change the model's advanced settings like temperature max tokens and whatnot now over here you have the new thread option from here you can create new threads then over here you have the hubs option you can install models from here another thing I like about this is that you can also see which models suit your system and which of them do not pretty cool you can also set up models by pasting the hugging face link over here now here's the local API server you can expose API endpoints for the model through here and you can also check logs here as well you can also change your API endpoint URL if you wish to now let's go to the settings option here the first tab is my models here you can see the installed models and you can also upload your own ggf models if you wish to through this option then there's the advanced settings here you can set up experimental mode which basically means that you can test out their beta and experimental features then there's the GPU acceleration feature you can turn turn this off if you don't want to use your GPU or turn it on when you want to use your GPU then there's the https proxy option here you can set up https for your API if needed next there's the extensions option here you can add extensions or disable extensions now here you can set up Gro API settings and then in this option you can set up open AI API Keys let's set up the gro API key and check it out with Gro gr's API key is free but it's rate limited so make sure you use it accordingly it is better to use it if you're not going to spam it with messages anyway if you don't have a Gro API key then go to gr's website get yourself signed up and go to the API Keys option and create a new API key then enter the AP AP key name and copy the API key now paste your API key here and that's it go back to the chat page create a new thread here and now in the models dropdown go to the online models option and select Gro llama 3 from here if you configure open AI API then that will also be shown here now let's send a message over here hm as you can see it's pretty fast you can use it with Gro open AI or any other llm provider you want I prefer local models but when I need super fast speeds I can use Gro pretty cool now let me also tell you how you can do rag with this to enable rag features you'll first need to go to settings and then go to advanced settings now over here enable the experimental features now rag should be enabled go back to the chat option and here you can see the attachment icon now first over here enable the retrieval option now attach the file here and ask the questions you want to know about the document or PDF pretty cool this is going to be really great for people who just want to get their models up and running without the hassle of the terminal or any other configuration it is really easy to set up via just one click install another thing I like about this option over others is that you can set up all your llm providers at one place whether it be Gemini open AI Gro or local models you can easily use it without any issues by just toggling the models in the drop- down menu it's really amazing I really like this one let me know if you will use it in the comments also so if you liked this video consider donating to my Channel Through the super thanks option below this video also give this video a thumbs up and subscribe to my channel I'll see you in the next video till then bye [Music]

Channel: AICodeKing

Views: 5,122

Keywords: jan, chatgpt alternative, perplexity alternative, rag

Id: gmCrnvOX5WM

Length: 9min 26sec (566 seconds)

Published: Fri May 10 2024