DeepSeek Coder AI 🐍 The Best Coding Model I've Tested? (Open-Source)

oh my goodness look at that perfect first try deepsea cater is the best open- source coating model out there it's better than GPT 3.5 and it's incredibly close in performance to GPT 4 so is it actually as good as they're saying we're going to test it today let's go so before we jump in I've created a new rubric just for coding models and so I have a bunch of tests that we're going to use today I'm going to evolve this over time if you have any suggestions of other coding challenges that I should give these models let me know in the comments below and thank you to everybody who commented on my Twitter thread giving me some of these tests that I'm going to be using today so this is deep seat coder it is a brand new completely from scratch foundational model let's take a look at it before we actually test it out so it was tested with 87% code and 133% natural language in both English and Chinese it was pre-trained on 2 trillion tokens and they have various sizes of the model from 1 billion to3 3 billion parameter versions it has a 16k window Conta size now some of these newer open source coding models are already being used as GitHub co-pilot Replacements and so you can have your very own GitHub co-pilot completely open source completely local no internet required running and visual studio code for free if you want a tutorial on how to get that set up let me know in the comments below and it also says it's open source and free for research and commercial use here's a little graph of the performance versus GPT 4 GPT 3.5 turbo and other coding models here's code llama and as you can see deep seat coder base performs better than any other open source model on this list now the one that they did not mention is the find model pH i n d and I'm going to be testing that in a separate video and here it is against the Clos Source models it beats GPT 3.5 turbo and it gets really close to GPT 4 in performance all right enough talk let's test it so I'm just going to go ahead and test it through the website itself and all you need to do is sign up for an account and you can test it but again this model is completely open source you can use one of the many methods that I've shown in previous videos for how to get this loaded on runp pod or locally on your computer so I've already signed up here it is and it has this pretty hideous background generated by AI so that's really cool that it was generated by AI but jez this background is very ugly all right the first test that we're going to run is one that we've actually used on every other test that we've run on our original llm rubric so create Json for the following there are three people two two males one is named Mark another is named Joe and a third person who is a woman is named Sam the woman is aged 30 and the two men are both 19 all right let's see here's the Json representation of the given data so we have people name name name gender and age perfect yeah that's an absolute pass and I'm going to start a new conversation each time so that it doesn't have any memory from the previous result and it won't cause any issues in future results next write a basic contact form in Python using any front end you want the contact form should include name email and phone number it should save to a database and one thing I'll say is it is extremely fast so looking at the code already it looks like it's going to use flask and SQ light all right here's the entire python section so let's go ahead and copy that switching over to visual studio code I'm going to open a new window I'm going to open a new tab paste in the python code it already recognizes it's python as main.py then we have a second HTML which is going to be the front end so I'm going to go ahead and copy the code there switch back to VIs Visual Studio code I'm going to create a new file I'm going to paste in the HTML and I saved it as form. HTML and it actually says save this HTML file as templates index.html so I actually need to change that I'm going to change a couple things here so it says main.py dop obviously that's wrong I'm going to go ahead and remove that second. piy I'm going to create a new folder called templates and then I'm going to name this index.html and put that in templates there we go now back to deep seek it says to run the application you will need to install flask so let's go ahead and copy this code and install it switch back to visual studio code let's open up the terminal let's go ahead and spin up a cond environment cond create DN form python equal 3.11 all right then we're going to activate cond cond activate form okay we're in form now now I'm going to try installing it using python from the folder I know it's already in all right there it looks like it's working and then we're going to run python app.py all right it looks like we got it up and running now so running on let's open it up and there we go we got a contact form with name email and phone number and I filled it out let's click submit oh we got an error no such table contact and I guess this makes sense because we never actually created the table so let's try to get it to do that all right so I'm just going to type when submitting this form I get this error and I paste it in the error yep and it says it doesn't create the table automatically so to create the table you need to run the following commands in your python shell so let's go ahead and do that so let's open up python then we're going to do from App import DB contact no module named app all right because I named it main.py let's go ahead and rename this file app.py and let's try again so from App import DB comma contact and that worked and now we're going to create all from the DB so db. creatore all and then parentheses looks like we ran into another issue so I'm going to copy these logs and paste it back into the app and see what it says all right let's try that okay maybe that worked let's try here we go python app.py let's open it up here it is I'm going to input some information and submit okay I think that might have worked I think it inserted into the database successfully all right there it is I opened up DB browser for SQ light and there's the information right there that's what I just submitted so that worked a couple of issues but pretty minor overall and really the code worked it was just the fact that I didn't have the database created so I'm going to give this a pass next write unit tests in Python for the following business logic ask the user for their name confirm it doesn't include anything but alpha numeric characters then reverse the string and output put it all right here's the unit test using the unit test library and it also gave me a description of everything so let's test it out copy the code okay so it wrote unit tests but now how do I actually tell that the unit tests are working I'm going to have it write code that completes the unit test and I'm going to test it myself so now write code that passes all of the tests you just wrote all right there it goes let's copy this code switch over to visual studio code now I have the unit test and I'm actually going to rename app.py to be test.py and then I'm going to create a second file paste the new code in there and I'm going to save it as app.py now one thing I just realized is how is it going to actually test that code let's make sure that the tests that we just wrote are related to the app.py file okay the tests were placed in test.py and the code to test is in app.py make sure that the test.py is actually testing the code in app.py okay so we import reverse name from app and let's give it a test okay looks like we got three passes so that worked now let's just make sure it's working so I'm going to run the code and see if it actually works so enter your name Matthew and there it is that's my name reversed perfect and so that's a pass for both the code writing and the code testing all right next we're going to use a challenge from the website a to core.com and this is a test in which we're going to have the model find bugs in the code so we have a bunch of code right here I'm going to highlight it all I'm going to go to deep seat coder start a new conversation I'm going to paste it in and then at the bottom I'm going to say find all of the bugs in this code okay so interestingly enough it says that it seems to be correct and there doesn't seem to be any obvious bugs however there are a few points that you might want to consider and it gives me a bunch of efficiency recommendations and then finally it just stops so this is a definite fail especially because it just seems to Output the same thing over and over again and for those of you who are wondering when I click go code Pier we have the errors right there so I hope this was a good test but it definitely failed okay next we're going to do a coding Challenge from the the website edit.com validating a set in the set game so we have a detailed description a bunch of examples and then we're going to check our code and what the model needs to do is right here write a function that determines whether three cards constitute a valid set and all the rules of being a valid set are right here as well as some examples so I'm going to copy this whole thing and I'm just going to copy paste it right into the model so let's start a new conversation paste and then let's go okay that's really short code but let's see if it works okay back here I realized it's expecting this in JavaScript so let me do that again but I'm going to specify that it needs to give me JavaScript so I said rewrite the code in JavaScript and here we are okay copy code let's go back let's paste it in and check all right test passed awesome look at that so that is a pass all right next I'm going to have it describe what some pretty complex code does and I've renamed all the variables so it gives no hint as to what it does but this is a quick sort algorithm so you can see it right here so I'm going to send let's see what it does all right there it is the given code is a python Lambda function that implements the quick sort algorithm here's a breakdown of the code this is extremely impressive so this code is a recursive implementation of the quick sort algorithm this is phenomenal that's an absolute pass all right now for the one I know you've all been waiting for write the Game snake in Python now deep seat coder uses this exact test on their homepage as an example example of what deep SE can do so I hope it's able to actually accomplish it in one go all right it's starting to output and it looks like it's going to be using py game okay that's really cool I've never actually seen this type of formatting before where it has an equals with a slash through it and I guess that's does not equal maybe that's just the formatting that it gives you but I I've never seen that before that's really cool all right it's done let's test it out switch over to visual studio code let's open up a new window I'm going to paste in all the code in there let's save it and it doesn't look like there are any errors although the pame game module cannot be imported so let's go ahead and install that pip install P game all right let's play it oh my goodness look at that perfect first try the score works perfectly all the directions work look at that I am so impressed one go and let's look what happens if we exit the screen all right and it stopped it didn't just quit it told me what the score was it waited and then it quit phenomenal absolute pass extremely impressive all right that's it for deepsea COD it's an extremely impressive coding model if you liked this video please consider giving a like And subscribe and I'll see you in the next one

Channel: Matthew Berman

Views: 62,308

Keywords: llm, coding ai, coding llm, coding model, programming, open-source, deepseek, deepseek coder, ai, artificial intelligence

Id: POSOmtJCOeI

Length: 10min 37sec (637 seconds)

Published: Fri Nov 17 2023