Replace Github Copilot with a Local LLM

hey everyone I've been testing the M3 Max and RTX 4090 lately and one of the key features I've been exploring is their ability to power through AI test now for most of us generative AI falls into two main camps image and text image generation is super important and I'm going to take a dedicate to look at it in a different video for today though I'm a coder and all this testing has got me thinking can I use local LMS to replace GitHub co-pilot we'll start our exercise by downloading the free LM studio and once installed called grabbing one or more code Centric chat models for reasons we'll see shortly I strongly prefer dolphin mixt so I'll type that into the search window from the results I'll use the UI hints to choose a quantization version that fits within my machine's Ram budget now this whole Ram thing is important so let's expand on it for just a moment for local code completions we want to balance speed and quality lower quantization typically means faster processing but gives lower quality results un fortunately for my testing most consumer grade gpus aren't capable of running models that are outright better than co-pilot notable exceptions however are the 4090 with its 24 gigs of RAM and apple silicon Max with that much RAM or more in short if you're buying a new machine in plan on running local AI juice up that Ram spec as much as possible anyway once we have our model simply click the server tab select the model and click Start server next in vs code we want to install the continue extension click this icon and press the gear icon and at the top of the Json config file add a new block of code in the model section the key here is to make sure the provider is set to LM Studio the other two parts you can just literally put whatever you want as long as you can identify the model in the UI and that's it local completions are ready so let's compare them to co-pilot starting with continue we use the shortcut command shift L to trigger an line prompt if no code selected we'll describe what we wish to create or if we have a selection how we'd like to edit that code one major downside of continue in its current implementation it doesn't yet support inline code completions like cod pilot now for many this will be a deal breaker and that's totally understandable the good news is this feature appears to be in the works so stay tuned on that front now for copilot the experience is well almost identical the only difference being those inline completions in the keyboard shortcut used to trigger inline chat which for co-pilot is command I so let's try a few coding tasks we'll start by asking it to write a function that adds two numbers next we'll ask it to change this to joining two strings and now let's revert this and change it back to adding two numbers and finally we'll do what's arguably most helpful we'll have it tighten up our code and check for common errors and prevent bad input now now as you can see with the exception of speed the results and experience are actually really close Okay so let's ask them to do something more interesting it's absolutely critical to note something here most models I try fail when you ask it to do everything at once however when you break Tas into smaller steps and build up it works quite well so for example create a web page that contains a blue circle the ball should be centered at the top of the screen horizontally and placed at the top of the page next we'll update this page so the ball drops to the bottom of the page using a realistic gravity equation when the ball reaches the bottom of the screen it stops finally update this logic so when the wall reaches the bottom of the page it bounces up realistically like a rubber ball on each bounce it loses some momentum until it stops completely finally let's try one more more create a web page add a file upload field to the form when a user uploads an image the image should be flipped vertically and displayed to the user so here's my takeaways dolphin mixt is really good it's not as fast as co-pilot but the results are absolutely worth it second convenience Pilots always there so long as you have a network connection local means starting a server eating up some resources and managing models still local is free and the results are currently much better if you can run the right models and that takes me to the last point the right models see dolphin mixol is a variant of Mixr with Mixr being the first model I've used that's Head and Shoulders above everything else when it comes to task based completions that is what we did here today the downside is Dolphin mixol isn't going to run well on most PC Hardware The X Factor is inline completions which again aren't yet supported on the continue extension this is important because in my daily use I rarely do what we saw here today I'm much more interested in completions and it's totally possible that more modest models will match or exceed co-pilot in that regard now if they do well that's going to be really interesting to see where the space is in the next couple years i' basically put it this way would you pay a cloud service for spell check anyway thanks so much for watching and if you're in AI image gen stay tuned for my next video

Channel: Matthew Grdinic

Views: 118,889

Keywords:

Id: F1bXfnrzAxM

Length: 5min 42sec (342 seconds)

Published: Mon Jan 29 2024