Copilot Causes Code Churn? This Study Is Concerning...

new GitHub co-pilot research finds downward pressure on code quality this is some legit research that just came out studying different projects and companies that have adopted co-pilot to see how their code quality has been affected over time and the results are actually really interesting so let's dig in new research on the effect of AI powered GitHub co-pilot on software development cites some adverse results the coding on co-pilot white paper from git clear sought to investigate the quality and maintainability of AI assisted code compared to what would have been written by a human in other words is it more similar to the careful refined contribution of a senior developer are more akin to the disjointed work of a short-term contractor I don't like this spectrum so to speak I feel like AI code's a secret third thing where intent can't be understood like I understand the intent of a short-term contractor they're just trying to get the thing to work I understand the intent of a senior developer they're trying to feel good about their job and brag but this in between that they're suggesting is where AI lives I don't believe in I think AI is this weird third thing of an attempt to solve the problem that you may or may not understand with implications underneath you almost certainly don't so we'll see where this goes the answer to this is summarized in this paragraph from the white paper abstract coding on copilot 150 million lines of analyze code and projections for 2024 github's quote is 55% faster coding 46% more code written 1.5 trillion added to the GDP numbers like these little wonder that github's own CEO has been taking time from his usual CEO duties to write about the AI Revolution apparently more than 20,000 ORS are already using GitHub co-pilot for business checks out you know what let's run a poll in chat quick I actually curious so it looks like that we're hard leaning yes with about 30% saying no for me it would be like yes but not anymore would be yes indifferent like if you weren't indifferent You' just pay for it my answer is always use it I I I'm a co-pilot guy now I didn't think I would be I even have multiple videos where I say like I did not think I would like co-pilot yet here we are and I use it all the time now so yeah yeah more no but interested on YouTube very interesting like there is actually a meaningful difference between the way the votes are skewing between platforms I've gotten the answers I was looking for here looks like between 60 to 70% of people who hang out in a community like this one are already using co-pilot and at least 40% are actually liking it really good results I'm going to end the poll I have everything I wanted to see thank you guys we can see why so many people are using co-pilot for business because once you're using it on personal stuff it sucks to not have it for work which has resulted in a lot of companies adopting interesting a separate study that GitHub undertook by Wakefield in 2023 June 23 they assert that 92% of us-based velers working in large companies report using an AI coding tool that feels like to me okay there's the difference 90% have used an AI tool either at work or in their personal time that's a huge difference they did not include that in the article there or in the uh report there but in their personal time they've tried an AI tool that makes more sense to me the problem with AI generated code are they really embedding tweets inside of actual white papers that feels weird to me every time somebody gives me for a fake tweet in a thumbnail now I'm going to cite this say look there's actual research search for people site tweets too yeah I feel like my job is less cringe now that I've seen this anyways do as I say not as I do if you really care about these things go read the research and all the things it sites I'm reading three abstraction layers forward because I just want to know what's going on so let's read the abstract we find disconcerting trends for maintainability code churn the percentage of lines that are reverted or updated less than 2 weeks after being authored is projected to double in 2024 compared to its 2021 pre Baseline we further find that the percentage of added code and copy pasted code is increasing in proportion to updated deleted and moved code in this regard AI generated code resembles an itinerant they're just making up words at this point I see why Prime didn't read this article though this regard AI generated code resembles a contributor that's whatever this board means prone to violate the dryness don't repeat yourself of the repos visited yeah AI is anti- dry for sure AI doesn't care about what repetition even is AI loves to repeat in fact AI was basically built to repeat at scale that serves as a Counterpoint to the findings of some other studies including one from GitHub in 2022 that found for one thing developers who use GitHub co-pilot complete task significantly faster 55% faster than developers who didn't use GitHub C pilot I will say I lean in favor of that I have one little bit of proof this is my proof I've never Broken Top 100 in Advent of code before I can break top 1,000 but for me to be 256 day one having lots of sub 1,000 times it's really really nice being able to write a comment to name a function such that co-pilot knows what to do from there lets you move stupid fast like stupid fast I don't remember what the day 24 was let me double check it so I can see what I did oh it was a position and velocity thing yeah I was able to just write the start of the function be like hey this function should do this and then co-pilot filled it out for me and then I got an answer before anyone else well before hundreds of thousands of other people oh I I have confessed to using co-pilot for I of code many a time that's just reality like they say not to use AI tools what they mean is don't use things you like pass the question question input to and then get an answer from I'm using co-pilot if I get banned from Avent of code leaderboards for that cool that's a that's some good content I can make back to this so as I was saying with 55% faster number I believe that I can go pretty stupid fast with co-pilot for a lot of things doesn't mean I trust the code I wrote with it but I can run the code and I can get answers it's really really nice and I just saw a question I'll Sidetrack for a little bit is co-pilot the best codei I don't actually care if it is or not because co-pilot has the best DX at integration there are a lot of things that claim to be 10 to I don't even know what % better than co-pilot in terms of the likelihood you accept the recommendations it makes but I don't care about the quality like Da Vinci resolves a better video editor than Final Cut but Final Cuts experience is the thing I like and I think we'll see this with AI tools where everyone's so focused on making the the best performing AI tool with the best results and not on how we actually use it doesn't matter if you make something twice as good as co-pilot because people will still just use co-pilot anyways the study was noted in the new white paper from git clear which sells a cloud-based code review tool interesting that git clear is also doing code review type stuff so they have they have an incentive to go after GitHub a little bit here it's a very good call that they made here about that cuz I had no idea what get cleared did in addition to productivity the GitHub study also measured positive effects in developer satisfaction in conserving mental energy git clear's research however investigated how the composition of code changes when AI is used get clear said its report sheds light on the following first what are the three significant changes since co-pilot's introduction two what do technical leaders need to be on the lookout for in 2024 and three how can you measure the impact of AI on your team's code quality regarding the first item the paper indicated the three most significant changes associated with co-pilot's rise concerned churn move code and copy pasted code buring churn the bottom line is that using co-pilot is strongly correlated with mistake code being pushed to the repo this is something I don't disagree on I know primes had a lot to say about this where as much as co-pilot saves him time he ends up looking at the code that was written much later on when there's a bug not really understanding it because he didn't write it or really review it it just appeared in his editor and he committed it and I you know that's been my experience too when co-pilot does its thing I don't think about it enough especially for more complex functionality when it comes to like markup or names of components or calling things like a used query it's really nice and have to type those things out or defining a type but when it comes to complex logic it's a little too tempting to hit that tab button and once you do that You' just accepted code that you might not be fully familiar with and that results into a pretty significant bump year-over-year in code churn my counterargument here for why these numbers are going to be inherently kind of is that the companies adopting AI tools are going to skew more I to put other than like hypergrowth like companies that are more interested in growth moving fast and doing all sorts of crazy stuff those same companies are going to be the ones that are hiring more grabbing more Junior developers and possibly churning more code inherently and having more problems too I would argue a bunch of Junior devs being hired by your company would have a similar impact to all of your employees suddenly using co-pilot in terms of these specific metrics so unless they're controlling for that that I'm not sure how reliable comparing AI companies to non companies necessarily is in this way we'll keep digging though I'm sure they did a good job of this the next part here is less moved code implies less refactoring and less reuse combined with the growth in code labeled copy pasted there is little room to doubt that the current implementation of AI assistance discourages code reuse instead of refactoring and working to don't repeat yourself code these assistants offer a one keystroke temptation to repeat existing code yep again it's the temptation of the tab that it's just the code is there you could write it or you can hit Tab and not have to it's just sitting there waiting for you to do it the same way your phone is there and you see the notification and you don't want to click it but you want the bubble to go away so you just click it anyways and now you're on Twitter for 20 minutes it's that but for our jobs which is very scary and then this third Point here is that more copy pasted code implies future headaches yeah there's perhaps no greater Scourge to long-term code maintainability and copy paste the code in effect when a non-keyword line of code is repeated the code author is admitting I don't have the time to evaluate the previous implementation by reading the code instead of reusing it the chore is left to Future maintainers to figure out how to consolidate parallel code paths that Implement repeatedly needed functionality I got a spicy take here this has been a problem for a while nothing to do with AI or even copy paste Cod genen has this problem really bad I'll never understand why everybody loves the crazy code generator helpers that stub out 15 files for you to make a Hello World with rails once you've got your project started everything should be pretty easy to do by making a new file and writing code and if it's not I would argue it's a failure of the technologies that you're using I have concerns that in a lot of ways AI almost become a replacement for code gen and people are doing the same thing they were with code gen where they were adding a bunch of code even though they only needed a little part of that functionality be it they copy pasted a huge thing from somewhere else AI copy pasted a huge Chunk from somewhere else or a code gen tool stubbed out 15 files when they only need two all of these things feel like the same problem to me just different ways to get there someone said isn't that basically what creat T3 app is absolutely not in fact we went out of our way to make sure once you've run create T3 app you can't run it again it's done you have started your project if you want to add a new route to your project you have to go make a new file it's not part of your developer experience it's part of your initialization process which is fundamentally different if the rail C is just a way to init a project I wouldn't give a when you're going to add a new route to your rails project people always reach for the rails CLI because it just does it for them and there's so much Boer plate they have to write again Cen is scary to me and co-pilot and tools like it take the Cen problem and make it one tab press away constantly it's a lot of small versions of the same issue and it's interesting that this study is making me realize that all these problems are kind of the same problem the paper concludes how will a co-pilot transform what it means to be a developer there's no question that as AI has sured in popularity we've entered an era where code lines are being added faster than ever before the better question for 2024 is who's on the hook to clean up the mess afterward I could do a very long video about this question in my opinion the future is that when things get messy enough there they're not worth maintaining they'll be replaced with better simpler things by developers who are probably using the same AI tools just building it better but if one codebase or one company's code falls apart because they didn't have good maintenance processes throughout now they're worse at addressing their users needs another company can form get good enough much faster and then compete with them cool this article listed a bunch of other studies that have been done about GitHub co-pilot and its effects on developer productivity and code maintainability over time very interesting to see how much this has been studied I had no idea people were digging this deep on it let's see what the conclusions are first one's conclusion was we found evidence which corroborates the current consensus in the literature co-pilot is a powerful tool however it should not be flying the plane by itself second study's conclusion was our empirical analysis showed that GitHub copilot's a promising tool based on the results however further and more comprehensive assessment isn't okay useless third one as more developers Embrace these tools and acquire Proficiency in the art of prompting with generative AI it becomes evident that this novel approach to software development has forged a unique and extricable link between human and artificial intelligence the symbi iotic relationship has the potential to shape the construction of the world software for future Generations that's a very bullish take final one the results of This research indicate the developers opinions are divided most of them met GitHub co-pilot before attending the survey the attitude to the tool was mostly positive but not many participants were willing to use it concerns are caused by security issues associated with using who the did they study this study is very interesting to me why am I not surprised it's Poland no offense to polish devs but y all love maintaining the pass for the purpose of the study a total of 42 answers from developers with different seniority and specializations were examined 42 people how did they find them the survey was created in Google forms and distributed between software devs from the author's environment and through social media including Facebook and Linkedin I can't say I'm surprised that Facebook and Linkedin devs taking Google form surveys are a little more skeptical of AI tools than other Developers for some reason that doesn't surprise me so yeah per always read the methodology of your research before you cite it because a lot of the time it is I what I have to say on this one I still use co-pilot I still love it I'm curious about you guys though do you use co-pilot do you like it you're allowed to use it at work if not why let me know in the comments I do really want to understand what people's experience what these tools are like because I was skeptical going in and I came out a huge fan so let me know chat with you guys later peace nerds

Channel: Theo - t3․gg

Views: 75,431

Keywords: web development, full stack, typescript, javascript, react, programming, programmer, theo, t3 stack, t3, t3.gg, t3dotgg

Id: 7ktvyqvWkiU

Length: 13min 21sec (801 seconds)

Published: Sat Feb 03 2024