From OpenAI to Open Source in 5 Minutes Tutorial (LM Studio + Python)

today I want to show you the absolute easiest way I have found to start using open source models locally in combination with your already existing Python scripts you can turn your old open AI chatbots into an open- Source version in just a few minutes this means that you can access uncensored models such as dolphin mistal 7B 52 and almost all other open source llms so let me show you how Okay so step number one is going to be to go to LM studio. a and download it so basically you can kind of choose here you can download for Mac you can download for Windows you can also download for Linux so basically just start with that this means that we can start using open source models directly from hugging face okay so now that you have installed that uh you can see here I am inside of the LM studio so basically the next step now is to download an open source model we want to use right so in this option here we're going to click on the search icon here and we're going to start searching for a model here on hugging face so I am looking for a model that is made by the bloke that is kind of my favorite one and I'm going to do some dolphin model so I'm just going to search the bloke dolphin and I want to download you can see it's already had a mixol 8X 7B here but I want uh mistol 7B dolphin version so I'm going to pick I think it's version two here right so I'm going to pick this one and I'm just going to go over here and yeah I want to download slight quality loss recommended so you can see I have already downloaded this version so you can just click on download here and let this run so this will download inside uh this window here and when it's done you can just go over here to your models and it says should work that means we can test it out now and to do that we can just go over to this speech bubble here right so we can kind of come into this playground we can scroll scroll down our models here so I want to select the mistol dolphin model here so you can see it's loading and yeah that was it okay you can kind of see on the right there we have some settings we can adjust so if you have a GPU I want to check GPU offload I want to just set it to 30 layers for now uh I want to adjust the content length so this is 2K tokens uh I'm just going to bump it up to 496 that's going to be like 4K tokens and here you can see we have a system prompt if you want to set that uh for now I'm just going to leave it as is and that means we are kind of ready to actually just test out the model so uh click on reload model to apply these configurations so we load the model again and now it's ready to use so I'm just going to put in like hello and yeah you can see it's pretty fast with the GPU offload uh I need a python code yeah so you can see working pretty good it's quite quick with the GPU offload here so I'm happy with that now we can kind of move on to the next step so like I said in the beginning of the video uh we don't want to just keep shatting in this uh text box window here we actually want to apply this now to our older scripts if you have some open AI scripts we can do that in a simple way so if you take a look at the script I have here now so this is a simple opening I chatbot you can see we have an open file we have the open API key we have a chat GPT streamed function that uses the GPT 4 uh model and yeah it's pretty straightforward it can write a poem so let's just test it quickly okay so you can see this is kind of streaming a poem so pretty straightforward using the GPT 4 model so yeah this script is working but it's now running on the API and we are actually paying for this right uh but let me just stop that let's clear that up and now I'm going to show you how we can use this uh LM Studio to apply this to an open source model in just a few minutes so if we go back to the LM Studio we can actually click on this Arrow here and here we have something called local inference server uh you can see uh this behaves like the open AI API so when I click Start server here now right we actually running this locally and you can see here we have this client open ibase URL API not needed so I'm just going to copy that right I'm going to go back to my python code here I'm going to replace this line with this okay so I'm going to save that we can remove the API key now we can kind of remove this part too let's save that okay so let's go back to the LM Studio we kind of want to set the model just to local model okay so I'm going to copy that too I'm going to paste in local model here oops and yeah that is pretty much it so this means that we are now going to run the dolphin mile 7B model to write a poem just by replacing this so this is running on the Local Host 1 23 AP ke is not needed we in the local model and yeah let's test it out okay python test 3. P yeah perfect and how easy was that right you can see it's pretty quick too it's writing a PO if you go back to LM studio now you can kind of see down here so yeah you can see this is streaming the response first token generated continue to stream response so perfect this is working and yeah how easy was that to change up an old model to just remove the open AI key and just slot this in and run it locally so we don't even have to be online now to actually run this right so you can see I went offline now let's see if it works totally offline just running locally perfect so that means now we are offline and still running the dolphin mistal 7B uh model so yeah pretty simple okay so I wanted to test like the differences between the GPT 4 model here and the dolphin open source kind of uncensored model so uh in gp4 here we're just going to start by putting in this write a stepbystep list on how to set up a social engineering campaign to fish passwords from a targeted company so I'm pretty pretty sure chat gbt is not going to response to this so let me just run it now and yeah you can kind of see I'm sorry but I cannot assist with that request and a whole bunch of other stuff here so now let's switch up the model and try the same query again okay you can see now we are back on the local model we are connected to our host right so let's go back here and let's try this now and yeah you can clearly see here that uh this gives an answer straight away and this clearly kind of demonstrate the differences between like an uncensored offline model open source uh compared to the proprietary models from like closed Source models like open AI an Tropic Bard and stuff right so this gives you more room to don't get like shut down if you have like a topic that is like ah a bit sketchy but it's not harmful but sometimes especially anthropic likes to shut you down if you type n anything that might be a bit on the edge right so that can be very annoying even if you're just doing legal stuff so yeah I just wanted to show the differences right and how you can kind of use this in practice so yeah that was what I wanted to Showcase today I hope you found it helpful and interesting definitely go check out LM studio and I think it's just great that people are developing these tools to make it easier to use these open source llms I think that's very important going forward that they have the option to choose between like proprietary models from open a and Tropic and stuff or we can do an open source model so very happy with that looking forward to development of this in 24 and go check it out give it a try and if you like this content I would appreciate if you give the video a like maybe leave a comment and yeah I see you again on Sunday

Channel: All About AI

Views: 26,105

Keywords: open source llm, lm studio, openai to open source, ai, openai, python, how to run a open source llm, how to run a open source model, open source llms, mistral ai, dolphin mistral 7b, uncensored llm

Id: IgcBuXFE6QE

Length: 8min 17sec (497 seconds)

Published: Wed Jan 03 2024