Using ollama and phi3 in VS code as an github copilot alternative

hello dear viewers I hope you're doing excellent in this video I will show you how you can run Microsoft's 53 model as a free co-pilot inside vs code we will achieve this by using olama 53 model and the continue extension but before that let me introduce you to olama olama is an open source app that lets you run create and share large language models locally with a command line interface on Mac OS and Linux given the name AMA began by supporting Lama 2 then expanded its model library to include models like Mistral and Fu AMA makes it easy to get started with running llm on your own Hardware in a very little setup time you should use olama if you want to be able to run llms on your laptop keep your chat data away from third party services and one to interact with them via VIA a command line interface in a simple way you can create your account uh on ama.com sign in and this is the screen that will be shown to you so as we discussed Lama o Lama provides different sets of models including Lama 3 five Mistral GMA Etc right so we'll take a look at all this model libraries that are available but in the meantime what I would like to do is start the installation process of olama now I have U WSL setup in my Windows machine so I'm going to select the option of running it in Linux now please note at the time of recording this video oama for Windows is in preview mode when an app is in preview mode uh there are no uh shortshot things that the app will work out um in the best possible manner right so so be careful when you are installing Windows if it's in preview mode so I'll click on the Linux part and I get this curl script which will download the installation shell script and execute it okay so I click on this copy button and open vs code and I will paste this shell command Okay so the installation will start in the meantime I can show you uh rest of the part of AMA now um when I wanted to get started I was looking at the document section but it was not there and then when I scrolled or looked at glanced at the bottom I could see the doc section here so if you're stuck anywhere uh go to this doc section uh they have nice documentation for different um you know be it the Linux setup process or the AP Pi reference or um if you run into any problem um then you can also refer their troubleshooting guide so don't forget to access the U document section here um of AMA okay so uh if I just put hyphen Library I would get a list of all the models that are supported by olama so you can simply scroll through this list and you get all the list all the list of the model that are supported by it okay so let's say if I click on it I would get some description of it some architectural um details would be mentioned okay the license and the readme so before you use any of these models make sure to read the documentation of those respective models let us check the status of our installation okay so installation is complete now let's take a look at the F model developed by Microsoft 53 is the most capable and costeffective small language model it has proved to be a better model when compared with models of the same size and next size up across a variety of language reasoning coding and math benchmarks the model that we are going to use today is the 53 model it has 3.8 billion parameters and it is available as part of the Azure AI Studio hugging face and O Lama it comes into two variants 4K and 128k context tokens okay so um it can generate uh some really good output and it also has a larger context window the second thing is in the future Microsoft is also going to release two more models the 53 small and the 53 medium models there's no deadline set on when these models would be available but if you're looking this video at the time when these models are available you can change Mini model uh with the new models that have been released now let's talk about the vs code extension that we'll be using so continue is a open- source vs code and Jet brains extension uh it is an you can think of it as a close alternative to GitHub co-pilot it provides the following features autoc completion completes single lines or entire sections of code in any programming language as you type reference and chat you can attach code or other context to ask questions about functions files the entire code base and more highlight and instruct so you can highlight code sections and press keyboard shortcut to rewrite code from natural language so these are some of the highlights of this extension uh please go through this readme file um so that you understand better about this extension also you can integrate uh let's say you have a subscription to uh open AI models so you can also plug in the API key and use GPT 3.5 turbo or GPT 4 models uh with the continue extension and you'll only be paying for the tokens that you'll utilize with the open AI API keys there is no cost associated with the continue extension you only pay for the tokens you will use okay so that was about continue now before we get started with the setup I want to tell you that olama performs sign ific anly better if you have GPU or if your laptop Ram has 32 GB or more of memory now let's come back to vs code and start using 53 models inside AMA okay so if I simply type O Lama I would get uh the list of available commands that are there I will type O Lama and I will list out if there are any models available so there are no models as of now so what I'm going to do is I'm going to pull a model from the AMA model registry so to do that I would type AMA pull 53 model so this is a very large file and I would pause the video here until the downlo is complete okay so the F model is downloaded let's see okay so when I use the list command I can see it here now what I can do is I can make it run from this command line interface and the command for that is AMA run 53 so I'm going to ask a question uh what is the capital of Sweden Stockholm that's correct it's not the largest city in Sweden but also serves as the political and economic center now we are getting this explanation because uh 53 as we looked into the description is good at uh reasoning tasks and that is why it is giving me some um historical context why it is why it is the capital okay and yeah I think that's it I mean uh we we are able to run uh the F model locally but I will uh exit it by typing uh forward slash and The Bu command okay so we are out of this now the next step is to integrate this with uh the continue extension okay so what we want to do is uh go to our extensions tab here and type continue H so I'll install it okay so we have continue install now and this is from my previous installation so I'll just get rid of this okay so when the first time uh you'll be shown the continue screen it will ask you to execute certain commands uh there there was another command and it was to install the star coder okay so I will also install it it is used for uh code autoc completion okay so oama pull star coder or there is a shortcut I can also specify run and um it automatically gives me the star coder model okay so this also I will pull and when you run it for the first time maybe Lama 3 is selected but you don't have to use that simply click on this o Lama 53 latest model okay and you're good to go with this so the download is almost complete now buy okay uh there is a command that lets us check if the AMA service is running or not so I would check the status okay so it's uh up and running now let's see um if codium Will generate uh some code for us so when you see this um raise to I it basically means control I on windows so I'll press it and I'll ask write python code to read the CSV file using pandas okay and I'll hit enter now let's see what happens I'm not pausing the video uh because um I want you to see that it take some time for the code to get generated okay first L came in replace there is a comment from F model right it is automatically doing things on its own I'm not typing anything or pressing the Tab Key simply get rid of this okay so it has it is still writing the code but you can see that we can use f model to write code um it may not be as fast as GitHub co-pilot is um because it depends a lot on your Hardware Ram uh in my case it is quite low so that's the point I wanted to make that you can can use f model as well uh as an alternative to GitHub co-pilot um you can host it locally and run it it's not perfect the the code that it has written but at least we know that it can write and I can then accept all reject or you know select this there are some instructions right so I'll just accept it for now okay so I'll get rid of these things and alternatively you get some more um help here uh through your code basis so if I press add there are some commands like uh I can add some I I'll highlight I'll select this code and I'll press crl l so as soon as I do that first let me get rid of this and as soon as I uh click on uh contrl L this becomes part of the context now I can ask any questions so let's say uh modify this code to only read The Columns call One Call 2 and call three okay now let's see what it does anyway so this will take its time to complete and we can verify I mean I I can see that it is correctly um specifying The Columns that it should be reading based on the new instruction that I gave and that's all about uh running F model in inside AMA on your local machine um for you to use it as a GitHub co-pilot alternative and I would encourage you to check out this extension play play around with it uh try out different models if you have larger memory try out the Lama 3 Model it is much much better in any of the present competitors that are there now once this code gets generated I can move it directly to my uh cursor Okay so so what I'll do is I'll replace this entire code and I will select uh this part okay so it has replaced this code okay and now I just have to read it so that's how uh this extens works with the fire model so that's all for this video I hope I was able to make you understand how this integration works and how you can use f model inside of AMA as a GitHub co uh Co pilot alternative so thank you for watching this video I hope you have a nice day thank you

Channel: Saurabh Jain

Views: 3,985

Keywords: copilot, ollama, phi3, continue, vscode, developerproductivity, productivity, llm, oss

Id: UUfHKn2Ky-0

Length: 16min 37sec (997 seconds)

Published: Wed May 01 2024