I Tried Every AI Coding Assistant

okay so first up we have tab n which is sort of like GitHub co-pilot if you've used that if you haven't I'll be looking at it in a minute but the core idea is that within vs code or whatever IDE you use it can autocomplete code as you're writing it and we get this AI chat feature as well so let's say function and we can see it tries to autocomplete the function and let's say we want a factorial function which it will autocomplete and there we have it we have a factorial function that I didn't even have to write I wish this was on multiple lines but that's nitpicky it is a correct function and let's try the chat feature as well so let's say what is the time complexity of factorial okay so it says the time complexity of factorial is o n where n is the number of iterations which is correct but then at the end here it says the time complexity is therefore of N squared which is incorrect so it was correct and then it sort of lost the thread a little bit and said in squared and it was incorrect so close but this is not correct information but okay let's try to break this code a little bit and see if it's able to fix it so I'm going to make this equals instead of less than or equals and see if it realizes that this is not going to work for a negative input anymore so let's say the code for factorial is incorrect how can I fix it let's see if it's able to figure this out okay so it says the code for the factorial function is incorrect because the condition for the base case is incorrect the correct condition is if n is equal to 1 so that's the condition I have so it's correct that the base case was wrong but it didn't do a good job of actually telling me why it was wrong let's look at the other response it has and and okay so this one the response number two is actually correct it should be n is less than or equal to one instead of n equals 1 so I'll give it sort of partial credit for that out of curiosity this first one it doesn't look like I have multiple responses here but this one at least it gave me a second response and the second response was correct okay so now I want to try some of the other command options we have so let's say just explain the code and let's see what we get for an explanation of this code okay so I'm not going to read all of this but this is a correct explanation of how a factorial function works so good job on Tab 9 there okay so now I'm curious if it can write some comments for me so I'm going to come up here and try to write a JS St style comment and let's see if it's able to autoc complete this I'll remove this thing that was added by vs code okay so it's autoc completing something we have a parameter of n that is correct we do return a number okay and now we're sort of going off the rails a bit we have API private and we're just sort of adding random stuff now it was close it started getting it right and then it was sort of going off the rails a little bit but what if I just add a blank comment like this it just says factorial function that's not particularly useful I know that by just the name of the function it' be better to tell me things like what inputs it takes and what it returns all right so now what I want to do is see if it can generate some test cases for me so let's come down here and just say test cases and see what it generates below us okay so it generated a bunch of test cases factorial 5 120 0 1 2 3 4 and if I run this that is exactly what I get so these test cases are correct I'm curious also there was a test cases command here here so let's do that as well and see what we get so generate test for code and let's see what it does with that okay so it does basically the same thing and we got a few more test cases now what about a slightly more complicated function so let's say function let's do merge sort okay so it knows it takes in an array okay so it does seem to be a merge sort function but it just added this merge function and didn't actually give that to me so let's see if we can get it to add that as well so function merge it seems to know that that's what it needs to add okay so it seems to have added the function and let's add a test case as well so let's say merge sort test cases and let's see if it adds some of those all right so merge sort the first one is just already sorted the second one is in backwards order and it looks like it did sort those correctly and let's see if we can add some more test cases all right so that one just said merge sort of one and it didn't really give me full test case oh wait it started doing it and now we just have basically the same test case again I would like to see it just generate more random test cases let's see if we use the command and instead generate test for code all right so it did generate a few more merge sort test cases but these are basically the exact same so we have 3 2 1 10 n down to zero and then all ones but I'd like to see something with just like random numbers and random orders but regardless it was pretty close and definitely still helpful okay so tab 9 works but there's better options that we'll see later in the video so I'm going to put it in the decent tier for now and by the way in this video we'll be building up this tier list of all of these different AI coding tools all the way from the 10x developer tools down to the artificially unintelligent okay so next we have chat GPT but I'm not going to spend too much time on this because I'm sure you've all used it at this point and I've made a lot of videos on it but here's my thoughts on chat GPT as a coding tool as it stands right now so I do think it's an amazing tool and a lot of the other ones on this list actually use the GPT API but that said I don't think for most coding tasks that it's actually the best option it's great if you're trying to learn something because it gives you so much extra context and it's great at holding a conversation however if you are just just trying to work through some problem going back and forth between your code editor and chat GPT just adds a lot of context switching which I think can be a bit of a problem and this can still work but why do this when there's other options on this list that work just as well without the extra context switching and the other thing I don't like about it is that it doesn't really site any sources and it just confidently tells you incorrect information sometimes which of course isn't a great thing so on occasion you just get bad information and you don't know it's bad because it's not citing sources and it's so confident in the information it's giving you that said of course chat GPT is useful so I'm going to put it in the actually useful tier but there are still ones that are going to be going above it next have GitHub co-pilot so it's going to have an AI chat similar to chat GPT we can also do command I to ask questions in line or we can just use autoc completion similar to what we had in tab 9 so we can say function factorial once again and it's going to complete the function interestingly it changed the name to factorial Li but I guess that's fine and we say if the number is zero return one otherwise return num * factorial num minus one and this will work but it's not going to work for negative numbers so let's see if it actually able to fix that so I'm going to come here and simply say handle negative numbers and let's see what it does to fix this nice so it gives me sort of a diff here where it added this if number is less than zero return not a number which seems like a totally reasonable way to do this I think I would have preferred it just throw an error but I think this works as well all right now come over here and let's just ask it what is the time complexity of this code and let's see if it's able to answer that question all right so it says the time complexity is o in which is completely correct let's see if it's able to add a comment up here we tried to do this last time and it didn't really work so it wants to add something that's like challenge let's do JS do style and see if it figures out that that's what I want factorial lie a number okay so it's just kind of adding comments like that but let's see because we do have the option here to do the sort of inline command ey thing so let's say JS doc comment and let's see if it can just add that in for me so it was able to generate this JS do comment with the parameter and the return so if you did want to use some commenting syntax like JS do it seems to do a pretty good job of that all right so now what I want is some test cases so let's see we have some commands here as well so we can explain the code let's actually do that first and then we can after we explain the code do the test cases you can pause and read the whole thing if you want but this is a great explanation of how the factorial function actually works so now let's see if we can generate some test cases so let's go to/ tests and see what it generates for us awesome so it actually generated an entire file for test and it uses the just testing framework so whereas tab9 was just sort of adding some console logs this used an entire testing framework and created an entire test suite and these tests look pretty exhaustive they include negative numbers which is good and I like how we have a few options here so we can copy the code we can insert the code into this file or we can insert this into a brand new file so it's going to create this testing file for us which is super cool all right so now coming back to the main file let's try to do the same merge sort function again so let's say function merge sort and see if it's able to autocomplete this for us okay so it takes in an array and it did the same thing that tab 9 did where it added this merge function but didn't actually implement it for us so let's see if it figures out on its own okay so it does figure out that we need the merge function just like tab 9 did let's see if it's able to implement that function it is now let's see can we just get test cases for merge sort all right so it generated test cases so let's just accept this and interestingly it actually put it in a new file for us I probably had an option not to do that but I'm just going to copy this into the same file it's also interesting that this one just generated it with console logs whereas when I did it over here in the chat function it actually used just although I'm sure if I specified which I wanted it would have done that for me but we can run this and yeah these sorting seems to be working and these are pretty good test cases much better test cases than we got from tab n because they're just more exhaustive for example we have arrays with duplicate elements and elements in different orders instead of just having like 0 through 10 and 10 through zero so yeah I'm a huge fan of copilot I've been playing with this all day long and it's working absolutely incredibly I've been trying to find some examples to give you all where it's not working very well and I just haven't been able to find many I do wish sometimes it would give a little bit more context with its sensors I think chat GPT on its own does a bit of a better job with this but overall co-pilot is incredible as a tool for actually writing code more efficiently I think chat GPT is where it go to actually learn something and co-pilot to just code more efficiently so this is going all the way at the top in the 10x developer category all right so next we have Bard and I'm not going to waste a ton of time on Bard just like chat GPT because I'm sure you get the idea of how it works but here's my thoughts on Bard and Bard specifically compared to chat GPT so I spent a few hours giving the exact same prompts to Bard and chat GPT today and honestly I was pretty surprised that in most cases Bard gave basically the exact same answer that said there were some pros and some cons so the big Pro I saw with Bard is that it does actually site some sources which is a great way to get some further reading and just to know that the information it's giving you is correct but on the negative side it does seem to just be a little bit less accurate than chat GPT for example I asked both of them to generate a tic tac toe game using react and chat gpt's game worked perfectly and B's code was close but it didn't fully work it didn't change which player it was and it wasn't correct at figuring out who the actual winner of the game was so I'm going to put Bard in the actually useful category with chat GPT although it's definitely in my mind a notch below chat GPT within that same category I don't think there's any problem with using it but personally I would just use GPT instead of Bard okay so next we have Amazon code whisper which is sort of Amazon's version of co-pilot although it doesn't have a chat feature this time but let's try it out so let's say function and again the factorial function you can take any number it's not autoc completing this part yet but let's see does it autocomplete now okay so it does and we have if number is less than or equal to one return one otherwise return num time factorial num minus one it added a weird blank line at the end here but I'll delete that and this is correct and now let's see if we can get it to generate some test cases so let's say test cases and let's see if it generates anything okay so factorial of zero is one and then let's see do we get more factorial of one do we keep getting them factorial 5 I don't like that it's doing one line at a time but this is working and we can run this and this is all correct but so now let's do the same tests we did with the others and try merge sort so we can say function merge sort let's see if it's able to automatically do this for us so let's see if it's going to complete the function okay there it goes and again it added a blank line at the end which is odd and it did the same merge thing where it doesn't give us the merge function but that seems to be a recurring theme and this one though is not automatically knowing that now it's time for the merge function but let's try doing function merge left and right and there it goes okay so it added the rest of the merge function for merge sort and I don't know where that came from I might have typed that though now let's see if we can also get some test cases so test cases for merge sort let's see what it generates nice so this one actually generated a bunch of different test cases although again these are already sorted for the most part so great test cases but at least it was able to generate some and I'm curious again to see if this can write comments for me but I'm guessing it can't because I don't really have a great way to actually interface with it to tell it that that's what I want but I'm going to try once again to put the beginnings of a js. comment and see what it autocompletes to if it autocompletes anything at all which doesn't seem like it's doing anything so that's a no on a j. comment what about just a standard comment all right so that doesn't seem like it's going to generate so I'm going to do option C so I manually generate it write a function that takes in a number and returns its factorial so this is like a prompt for a question on a quiz rather than an actual comment so that's a bit strange as well although I did want to show one cool feature of code whisper which is that it has this reference log where it actually gives you an idea of where the code came from so for example it actually says that this was from a reference from this code Wars repository which is kind of cool to be able to see that and then to make sure that the code that's being used doesn't have some license that you can't actually use it so overall code whisper is fine but its only feature is basically just one of the co-pilot features and it seems to do it worse than co-pilot does it might be better if you're specifically using AWS features and it might be really good at autocom completing for those AWS specific features but other than that for just like General coding I don't think it's all that great so I'm going to put it in the acceptable category all right so next we have Source graph Cody which is pretty interesting so it's similar to tab9 and co-pilot and all of these it's got an AI chat and the ability to autocomplete code but what it also claims to be able to do is understand your code repository and to make suggestions based on the entire repository so we'll see how that works in a minute but first let's give it some of these same tests so let's add a factorial function okay so it knows it takes in a number and it autocom completed the whole function so if n is one return one otherwise return n * factorial of n minus one so this is correct again although this is not going to work for the case of zero or a negative number so let's see if we can actually just get it to fix this so we can add an inline question so let's say can you improve this function so I'll give it this open-ended question to see how it does and if it doesn't answer it I'll give it a more direct question to fix the cases it's not handling okay so it says here's a few ways the function can be improved use a loop instead of recursion to avoid call stack limits on large inputs okay that's not what I was looking for but that is a good way to improve the function and then we have use big int to allow for larg inputs I don't know that we need that but decent suggestion I guess use memorization so another actually good suggestion to improve the function and simplify the conditional logic I don't really like this I don't think this is simpler I think it's just less lines but none of these are actually getting at exactly what we want so let me get rid of this and let's say how do I add another question in here do I close this I guess kind of a conf using UI okay so let's say make this handle inputs less than one ask Cody okay so here's how you can handle inputs less than one and it added this check if n is less than one return one so this kind of handles what I asked it to do but this is just redundant because you don't need all of this it can just be if n is less than or equal to one and also if n is less than zero we probably don't want to be returning one we should be throwing an error or returning in a in but some of the other ones did that as well so not horrible but not a great suggestion either but now let's see if it can actually add a comment so let's start by doing the test to see if it just autocompletes the comment for me which has not worked on any of them but we'll see if it works so parameter Good Start return number and we're sort of lost there so sort of started doing a good comment what if we just do the standard comment syntax factorial. Js so not a very good comment there either but now let's try to do JS doc comment like this ask Cody okay so that took a very long time to actually come back and it wasn't able to actually do it it explained what a js. comment is but it wasn't able to actually add it in for me but okay let's come over to Cody chat now and let's say let's see what we can get with the forward slash let's do some generate unit tests and see if it can generate some good test cases for us awesome so I'm going to expand this a bit and it generated some just tests for us so I like that it used an actual testing framework and we have some reasonable tests interestingly it does know to throw an error on a negative in but it didn't actually do that in the function which is kind of funny so this test would actually fail but I guess that's actually good because that means it's going to catch its own air all right but now what I want to do is test this ability to understand the entire codebase so I'm actually in the codebase for a video you might have seen on this channel a few months back where I had some Fiverr developers build this sort of task management application so this is a task management application and what I'm going to do is ask Cody about it so first of all let's say can you show me the logic for saving a task so if I'm new to this codebase I might want to see where exactly is that logic for saving a task so let's see if it's able to actually find it within the code base okay so it says it read eight files and it wasn't able to figure it out so let's see what file was it actually read so all it read was this file I have opened which I'm probably not helping it by having that open and then a bunch of node modules files so it didn't even really look to any of the actual codebase but let's close this file to give it the benefit of the doubt and maybe go into something near the tasks let's actually even go into I think it's this task section that actually has that logic yeah so this is the one that actually has the saving task logic in it so now let's ask the same question but it will give it a little bit more context so let's say this is a task management react application find the logic that saves a task on the server okay so let's see what we got back so first of all what files did it even read so again it just sort of looked through the node modules and then it looked at this file that I've opened and one other file so this whole idea of it just looking through the node modules is not very useful but okay let's see what exactly it shows so we have used dispatch and save task from actions and handle safe task dispatch safe task so is there any way to actually find this in the codes let's see this is going to copy it this is going to insert it and this is going to save it in a new file so I don't see any way to actually find this but let me see if I can find this in here if this is actually the correct code yeah so as far as I can tell the code that it gave me back is not actually code from the repo so this is the actual save task function which looks nothing like this so it really wasn't able to actually find that and provide me any useful information on how saving a task Works in this repository okay so my thoughts on Source graph Cody first of all it's just sort of slow probably because it's having to parse all of these files and it doesn't even really do a good job at the thing it says it's made to do of actually parsing files it was mostly just looking through the node modules and even when I sort of spoonfed it the right file to look at it still wasn't able to find the logic that had basically the exact same function name as what I asked it to find so it didn't do a great job of actually integrating with the repository at all from what I could tell so this is a cool idea and I like that they're trying to do something different and this has been one of my main complaints about AI coding assistants is like they work in these small projects but the second you go work at Google and there's two million files with aund lines of code in each of them they're just not that useful anymore and this is trying to solve that but it just doesn't seem to actually be solving it in a very good way that said the code Auto completion was decent and the AI chatbot was also for the most part decent as well so I do hope they continue working on this and maybe one day we'll be at the point where it truly understands your code repository but I don't think it's there just yet so I'm going to put it in the acceptable category okay so next we have codium ai which is really cool because it's not trying to do what all of the other AI tools are doing so it's not trying to be a Do-it all code completion tool instead it's just very very good at testing your code and helping you improve your code so I'm going to test this factorial function we have and see what it's able to come up with okay so first of all it generated a test suite for us so we can see the configuration it's using just and we could actually change the testing framework if we want we can see the behaviors that it actually covers so we have like the Happy path the edge cases as well as some other behaviors and we can see each individual test case and we can actually ask it to change any of these or to run it to make sure they're actually working and we can actually see it's handling some things that my code doesn't handle so these test cases would actually find errors in my code which is amazing so for example it knows we need to throw an error if we get a decimal as an input but I will say it's not perfect either so for example here it says we should return one for a non-numeric input value but I think here we should probably be throwing an error so let's say instead of returning one make these throw an error so let's say that this is not a test case we like this is not the expected value we actually have and okay now it says it is expecting to throw an air so let's go to the code explanation as well okay so we have code analysis the inputs and the flow of the code as well as the output okay and we also have some example usage this is awesome this code explanation is amazing this looks like professionally written documentation so that is great and then code suggestion and actually I'm just going to hide the chat because it basically does the same thing in this case so here we have our suggestions so first the code should check if num is a positive number if not it should throw an error so let's select this and say we want to do that and let's see what else we have the code uses recursion to calculate factorial instead it should use a loop to avoid potential stack Overflow errors also a good suggestion it depends on your exact use case but I do think in most cases for factorial even though it's a sort of example we use for recursion a lot it actually makes more sense to just do it iteratively because you won't have that stack Overflow so let's select this as well the code should check if number is larger than 170 if so it should return Infinity because JavaScript cannot accurately represent larger numbers than 170 factorial so I don't know if 170 factorial is exactly the cut off but that's a very interesting point so let's say we want to do that as well and somebody down in the comments can let me know if 170 is actually the cut off of the max number in JavaScript or not but anyways next we have the code should check if num is zero if so it should return one as zero factorial is one so the current code actually does handle this but maybe with the other edits it wouldn't so let's select this and say we want to do that as well the code should check if num is in a in if so it should throw an err that also seems reasonable okay so let's prepare these code changes and see what it says to change awesome so it seems to have done all of that so let's apply this to the code and see what exactly we get here so now we have if the type of number is not number or number is less than zero or number is not an integer we are going to say input must be a positive integer so that's good except for this error message isn't perfect because it can also be zero which I don't think is considered a positive integer just an integer but maybe it is I don't know is zero positive I'm not sure then we have if number is greater than 170 return Infinity if number is zero return one and then we have the iterative code here as well so yeah honestly this is the best test generator I've seen its explanations are really good and it doesn't try to do more than it's capable of it doesn't claim to be able to autocomplete every function you ever write because it's just not made to do that it's made to improve your code and to write good test cases which it does a great job of doing so because of that it's going in the top tier at 10x developer now all of these tools have their own place and all of them are probably valuable to certain people but in my case I'm going to continue using chat GPT as well as codium Ai and GitHub co-pilot that's going to sort of be my AI developer suite and hopefully it will make me a better developer but if you want to be a better developer you should watch this video next that the YouTube algorithm thinks will'll enjoy and let me know if it's right

Channel: Conner Ardman

Views: 439,307

Keywords: coding, software, engineering, software engineering, sde, swe, software development, development, web, web development, developer, tech, frontend developer, engineer, javascript, front-end, front end, frontend, js, dev, code, programming, chatgpt, gpt, codiumai, bard, sourcegraph, ai, copilot, github, github copilot, tabnine, code whisperer

Id: TXtnFw9eDmM

Length: 24min 50sec (1490 seconds)

Published: Wed Nov 22 2023