How to run Mistral LLM locally on iPhone or iPad

in this video we're going to look at how we can set up an llm a large language model directly on our iPhone or iPad now I did not create this or Creed it to I think it's mik who shared this on LinkedIn but we're going to do a video walk through and showing you how you can actually set this up on your own device so let's get started the first thing that we need to do is install test flight um app through the App Store so if you go to the App Store just search for test flight you'll see it's by Apple you'll open that up and it'll bring up a screen saying ready to test jump back to the tutorial and the next page that we're going to be opening is the llm Farm website and when you go to that page click on install with test flat and you can accept that and install and once that's installed just open it up click next start testing and you set up so you can see on the left hand side you got a little panel with some settings we're going to come to that next after that we want to we can close off that Tab and we're going to come to hugging face to a URL here which is ml 7B instruct v.1 and if you scroll down and the model we want is ml-7 B- instruct v.1 D ggf so it's this one just directly in the middle and when you click on that then have an option to download it and it'll start downloading to your device you do need um over 4 gigs of uh space and about 8 GB of RAM once that's downloaded we're going to go back into llm file and on the left hand side we're going to open that dashboard we're going to go to settings models and we're going to add a model Now find that model that you saved to your files and click on that and it will install it onto the llm farm app now in the tutorial does explain that this will then duplicate that file so you can go and delete it from the fileer that you downloaded it to the next step is we're going to click on chats we're going to click on plus we we're going to select a model and we're going to select uh the mrol 7B that we downloaded in the prompt format we're going to change that section that has the prompt in the double curly brackets and you're going to replace it with the prompt details that's provided in the tutorial so you can simply copy and paste that leave those settings as they are change the prediction options to enable metal and mlock K I don't know what they do but apparently that's important and then just click add at the top now you should be able to um start a chat now so when you click on the actual model that will then start a chat now this runs completely locally on your device so you don't need to be connected to the internet so let's turn off the Wi-Fi now we don't have any data on this device so Wi-Fi is off airplane mode is on so this is completely disconnected from the internet and we going to now come back to the chat and we have an option to type in a message so we can say hello and the first time this happened to me as well it said failed to load model so what I needed to do was actually close that app and then open it one more time so once you open it the next time come back into the chat and it should worth so let's just say hi and the first time loading is a little bit slow you can see it there's model loading and in the background it's kind of bringing everything together I suppose but after the first load it is a lot quicker so we got hi how can we assist you today great um can you help me create a strategic plan for a nonprofit question mark certainly here are some key steps to create a strategic plan for nonprofit organization Define your mission and vision um access your assess your current position set goals and objectives so the quality of this has probably been said to around chat gbt 3.5 but the fact that it can run locally on an iPhone or an iPad without access to the internet is pretty amazing I just started exploring this I think there's so much potential so much fun that we can have with this the last point I'll just point out is if on the top right you click that little uh reset button and it just does the tick the loow the chat interface doesn't actually reset so if you're familiar with chat GPT or another chatbot normally that whole interface resets it doesn't it only it refreshes the model but for some reason the actual chat doesn't reset so when you click high again often then has to reload that whole model so it starts off a little bit slower but from there you can do whatever you like so give it a go and see what you think as I said all credit to mik I think for on LinkedIn who shared this but really I think this is just a glimpse into the future Google's released Gemini which has a smaller model that's going to work on phones as well so it's really exciting to see what next year has in store for us so give us to try as I said if you got any questions please feel free to let me know otherwise look forward to seeing you in the next video

Channel: Kyle Behrend

Views: 12,885

Keywords:

Id: 5QEDNZlDf-c

Length: 6min 5sec (365 seconds)

Published: Thu Dec 14 2023