Empowering devs with AI: How Shopify made GitHub Copilot core to its culture

[Music] please welcome Farhan th hey everyone my name is furhan and I want to talk about co-pilot that we use very happily uh at Shopify now what's interesting about thinking about co-pilot is that I want to produce a few guideposts and the reason I want to talk about some guideposts first is that in order to get the benefit of co-pilot in your engineering team you probably have to think much more holistically about your company and about AI because what you don't want to have happen is you introduce co-pilot and the rest of your company is not really thinking through its AI strategy and that can mean AI for employees AI for customers and partners so you do really want to make sure you're getting the benefit of gen everywhere and of course through co-pilot but I want to walk you through how we got there and so what I'm going to talk about are three things Shopify engineering and GitHub we're a long-standing partner of GitHub two how we think about llms at Shopify and of course the meat everything to do with co-pilot and engineering and so first off let's talk about Shopify so for those who don't know Shopify is typically the brand behind the brand we power you know the I think the world's best dtoc sites and usually if you are outside of Amazon and shopping online and you've had a great experience it's Shopify Thomas was actually wearing the same shoes I'm wearing um which are from vessie a Shopify Merchant and so anytime you're actually thinking through how we power all of this it's actually because of the great work our engineering team and product team and design team built and we power you know over millions and millions of merchants all over the world hundreds of millions of buyers and you know uh my PR team messaged me and said hey we're almost at a trillion dollars of gmv through the platform so pretty amazing that way we can build [Applause] together and just some fast facts because it's actually interesting when you're powering a platform like this and again we use you know GitHub for all of our all of our workflow that what we can build together and so the first off is actually we build an amazingly highly converting beautiful checkout and it's powered in an ecosystem such that we can do you know sounds like these small numbers right 5,000 checkouts a minute but it auto scales up to you know eight times that and what's amazing about that again is we use a lot of automation to enable what we call flash sales so you can imagine someone like Kim Kardashian dropping a new lipstick or Alo yoga dropping a new color or Taylor Swift dropping some merch they don't always tell us when they do these things on social media and so our infrastructure has to automatically scale up to support all of this amazing load and of course all throughout with an amazing beautiful experience such that you get your what you what you want to actually get from these vendors so really really cool infrastructure and of course some stats up there also on just how much we have to support and just because we're coming into the season probably the busiest time for internet retail Black Friday Cyber Monday right the Thanksgiving shopping weekend or week or weeks now because people tend to have their sales earlier and earlier and what happens with Shopify is every time we have a Black Fighter cber Monday the internet goes crazy our infrastructure goes crazy and what happens is we set a new Baseline for what's possible on the internet for some of our Merchants when we do a launch it ends up being the largest sale in Internet retail history and then the next week it's the largest sale again in Internet history and then the next week it's the largest sale in Internet history so these things keep going and we set a new Baseline just an example of what a flash sale looks like because people tend to get interested in this infrastructure kind of like nothing nothing nothing nothing huge spike in traffic and then it curls off from that one from that one Merchant so you can imagine this is just one of the millions and millions of merchants on the Shopify platform again scaling up and scaling down it's pretty nuts and amazing to see on the infrastructure side and again this is what I talk about in terms of setting a new Baseline this is from a few years ago so every time we have a new Black Friday Cyber Monday all of our infrastructure ramps up and then we have this little bit of rest fight over the December period where it comes down a little bit but then it actually stays up that high it's pretty nuts because we end up just literally saying okay cool Black Friday subber Monday of 2021 is now just Tuesday of 2023 yeah pretty nuts I want to talk a little bit about GitHub because of course we are a large GitHub customer have been for a long time we have about 4,000 folks in R&D what's interesting about that is not uh it's not only Engineers who contribute to GitHub at Shopify product product managers ux data they're all using GitHub for a lot of their workflow we have about 300 public repos right about 5,000 private repos it's kind of like slack channels right there's more of those than there are people in your company same with in Shopify lots of repos I think we retired or archived 2,000 repos just last week so we're always trying to uh make sure that we've got the most effective infrastructure there over 10 million lines of code we joke that we have the uh maybe not a joke we we do have the largest ruby code based in the world right and as we try to simplify maybe that's uh a good thing or and not a good thing as we you know try to delete lots of code as well we do about, 1500 deploys per day uh like I mentioned over 4,000 contributors and over 100,000 commits per month pretty insane and how much we use the GitHub infrastructure to get our work done so now I want to talk a little bit about llms so in order to then get to where we want to get to which is empowering and having high productivity in your Eng team I think you really need to think about the entire AI journey of your company and the first thing that we always think about is making sure that we are bringing llms and gen AI to everyone because the Panacea is actually having AI to your customers right everybody wants to build something and have ai show up in front of their customer so you can whether it's a chatbot whether it's an experience such that um it's it's magical you want to make sure that your employees have that experience also at work and for Shopify that started with our merchants we launched when chat gbt came out November 30th 2022 within two months we had product descriptions I was actually chatting with a merch before this glad and young and they were talking about how they use this all the time to actually enable uh very quick product descriptions so imagine that you are selling soap and now you have written down all of the interesting aspects of your soap product you can press a button and magically you can have an SEO friendly highly playful tone if you want product description if you are someone who could do maybe a dozens of these per day you can now do hundreds we actually have the moniker magic that we use for that so all over the you know Shopify surface area you'll see a little magic symbol and it'll allow you to do things it'll allow you to create email subject lines product headings all the things that you can imagine that could be magical for your for your Merchants again the reason that we think through this for all of our surfaces is so that we can ensure that our employees are thinking about these tools right our Merchants use uh AI for our help center again we look through all our documentation and allow them to easily search and we even have something called sidekick which you may have heard about two days ago when Sam Alman had it on screen at open AI demo day he talked about how we're working with them to build um the amazing sidekick which is like a co-founder for your business the second surface that we talk about is our buyers and again the first one is shop AI which again allows you as a buyer to find a direct to D Toc brand that you would love and of course again I talked to this Merchant outside glad and young then they use the inbox feature which is basically asking questions from buyus Hey how do I exchange an item do you have this in red are there any sales on and again all powered by gen and I know I'm repeating myself you want to make sure that you are thinking about this in your company because having gen in just one workflow likely is not going to succeed didn't want to leave our partners and third party ecosystem um out to lunch we have over 10,000 apps on the ecosystem and now even our Merchants can use AI summaries in their tools to determine whether they want to install one of these apps but in order to get all of these things you have to start with your employees this is where co-pilot comes in but this is where lots of tools come in we are pretty big adopters of tools and this is an interesting chart that just showed when the Gen AI Revolution started again after chat GPT everyone started predicting where will the productivity come from right if you look at this chart the first one was sales we think we can automate sales emails sales content the second one was marketing content and the third one which surprised me engineering right why would engineering be up there right in the idea and we again we're seeing it with co-pilot now but it was actually a little bit of a uh you know a puzzle for folks saying why would engineering show up there what we say to our employees is reach early and often for AI to we use these tools very regularly and early because again we want to make sure our employees are thinking about this and what I would tell you what I tell my parents is search you reach for an AI tool early and often because you are not sure what's going to work and what not going to work and you can see some of the things we do here whether it's co-pilot whether it's chat gbt or even mid Journey with our design team don't even think about co-pilot until you've thought about AI for your company I think I've said it enough and one of the things we do say is don't think about AI as a replacement for jobs think of it as a replacement for tasks that is exactly what it's doing we're not trying to replace Engineers we're trying to replace the tasks that some Engineers do let's talk about co-pilot so this is an email I sent to Thomas after he got the job in 2021 I highlighted two things here the first thing is the date right December 2021 this is a year before chat GPT and the second thing is I let it is all Engineers this is a real email I was like I want co-pilot I want it now and I want it for everybody and what was amazing about this was one and I talked to the GitHub team this morning uh Thomas did give it to me and all of my Engineers which is Thousands the GitHub team did not know that he did that that's number one and then two what's funny is in the summer of 2022 he came to Toronto where I live and he messaged me and said can I meet up with you and I kept trying to dodge him because I was pretty sure he was trying to get me to pay for co-pilot and we were using it for free so we an our happy paying customer but back then I was like I'm going to keep using this for free as long as I can and so I'll show you some stats on what the adoption was but why did we end up choosing co-pilot there are you know competitor products I meet with lots of startups you you're watching like Twitter X seeing newsletters you'll see lots of AI tools coming out especially for code completion and for engineering productivity for us it was a few reasons the first was we did not want to increase our trust Circle so at Shopify we think about who has access to our data our code is in GitHub because our code is in GitHub we are not increasing the trust Circle to include co-pilot right that makes it very easy for us to say let's try it that's number one number two is GitHub and open Ai and Microsoft like all these partners that are working together are innovating very very quickly you saw Thomas this morning talking about all the things that they are launching right we've been lucky to be early access on some of those and the curve of innovation is what you have to make a bet on not on what co-pilot has today you can look at you know other tools and say oh we can try this one we can try that one we can try that one sure you should you should try them but the bet we're making is that the curve of what GitHub is doing will enable us to have all of those features very quickly and so you know I have a great we have a great relationship with the GitHub team we always we we talk to them and say hey this company's doing this are you guys bringing it to to GitHub co-pilot and I think there's one other interesting thing people don't maybe necessarily think about which is sometimes in doing the Deep dive on some of these tools co-pilot has decided not to offer something because it actually doesn't work so for example right now we talk about generalized models right when you use co-pilot and autocomplete everyone's got the same co-pilot right we all have the same co-pilot and Thomas talked about this morning that potentially you might want to train co-pilot on your own repo and then ask it questions what's interesting is in a lot of cases by using a specific model on your repo it might actually perform worse than the generic model that's trained on all the open source repos and so in having those conversations with a partner like GitHub allows you to then make the trade-off as like do you want to do this it's going to be expensive to train your own model and it might perform worse versus talking to a startup who might say this is what you need that's one another one is um you know talking about having a co-pilot for PR reviews right in early investigations with GitHub they noticed that the automatic um the automatic descriptions and commenting that comes back from a PR review that someone might submit um might actually just cause them to do work that is not actually that important and now you're adding developer toil without maybe getting the benefit and so again working with a partner like that allows you to think through the second and third order consequences versus just saying let's use this AI tool let's give them our data and they may or may not uh it may not be in your best interest to do that co-pilot adoption at Shopify we have done a lot right after that December 2021 email in January we launched co-pilot with all of our Engineers a year later copilot for business is basically the way of saying that we started paying for it a year later in March we started using some of the tools you'll notice some of the words here are what was announced today because we've been in Early Access with GitHub for a while right the ability to have co-pilot docs co-pilot chat on github.com co-pilot chat in vs code all the things that you saw demo this morning we've been we've been playing with them and again what's interesting is you're starting to see which ones developers are gaining uh confidence in and how their perceptions are changing but then also which ones are not yet don't not yet have product Market fit at least in Shopify as they get better and so we'll talk through some of those but there's a lot happening here that shows you the pace of innovation that's happening and the Panacea which is making sure that your engineers are more productive so how should we think about this right a lot of this was this actually came up this morning uh with folks how should you measure uh developer productivity I up there ask McKenzie I'm kidding definitely do not ask McKenzie that's insane right they're consulting company what you should be thinking about instead is what matters for your developer community and if you look at the Spectrum of Art and Science there are people on both sides right there's the science let's track 50 different metrics and let's see if our engineering T is doing well and then there's the art are you building the right thing for your customers Shopify tends to be on the art side we really believe it that software development and Building Products is more like pathf finding right if you ever see like lightning strike you see like the ground and you have lightning at the top and it's trying to find a path to the ground that's what we think software development is it's not a straight line it is trying to find a path you're going going forward you're deleting code you're prototyping you're starting over you start walking towards the horizon and you decide you want to go that way that is not something that can be numerically assessed easily and so when I talk to companies and a lot of you know CTO call me and they say well how did you decide to use co-pilot you know I'll talk through some of the metrics we do track and we did find interesting but one of them is happiness so it's interesting I walk through some of these things but we have found that in doing the quantitative analysis of copilot is not going to tell you the whole picture but doing a qualitative analysis might and I again I talk to lots of companies some of them and again we don't track that many things you know have like three or four things that we track here and again I'll tell you the reasons we track those but I talked to lots of companies that track the full Dora space they use the different tools I have a list of I think 50 different startups that want to connect to your GitHub and give you statistics I'm not saying that's bad I'm just saying it might be interesting data but it might not be telling you the whole picture one story I do want to tell you I talked to a company that had I think 3,000 engineers and they track 50 different metrics they deployed co-pilot and the CTO called me and said we've been running it now for I think it was two or three months they saw nothing no change I was like okay but he said one thing and I said okay so I saw nothing in them numbers but did you run a developer happiness survey did you do any qualitative he goes oh yeah yeah we did like a we asked our engineers and they off the charts were saying they were happier at their job and happier using co-pilot so again we see something similar it doesn't have to be in the quantitative measurements it might just be in the qualitative measurements and what did I when I asked that CTO again 3,000 Engineers so you know they're paying for co-pilot are you going to keep it he said of course I'm going to keep it because the engineers are happier that is like you know happy Engineers less turnover more happy at their job getting towards the you know Solutions faster like all those things matter and so that engine that CTO still rolled out co-pilot even though they saw Zero on the quantitative now I would call I think on the no quantitative change but that's what he saw and uh still ended up rolling it out anyway so what do we look at so we look at two or three different things across our ecosystem of course we look at happiness that is one we do a developer happiness survey we want to make sure that uh we are getting a lot of the qualitative feedback and we do see high correlation with co-pilot and happiness but we look at some other things we look at PRS and again we're not get PRS per week to compare person a and person B and saying oh that person has 20 per week and that has 30 like it's more about how do we unblock somebody if they're stuck and so again we see some correlation with hey this person is stuck they may not have a PR this week let's talk to them and see what's going on we see high correlation with pairing I love Thomas's whole pair programming I'm a huge pair programming fan we believe and feel like co-pilot is a pair programmer for your engineers and should be treated like a pair programmer right a smart co-pilot right funny name a smart co-pilot to work with you while you're going through these problems that hopefully lets you stay in flow longer than you would have been if you had to go and leave the window and go search something else one of the amazing things that's happening and you saw it again this morning's demo is the ability to have like a co-pilot chat now in BS code so you're you know I used to work with a lot of Engineers who would cut and paste code go to chat gbt come back and now you're leaving and going back and forth in these workflows having these things in line highlighting code and then asking co-pilot questions all that is now just in flow and again staying in flow is the name of the game because we know it takes you know 20 or 30 minutes to get back into flow and Pa programming really helps you stay in flow so we look at pair programming we see some correlation there of course i' mentioned happiness um and we actually really think that this makes a big difference even if you are not seeing anything on the qual on the quantitative which again I'll call a little bit of on that one for us what we do see is those Engineers who use co-pilot do tend to have some some correlation with impact and I think the key thing to think about in correlation and I talked about correlation with PRS correlation with pair programming correlation with co-pilot is its correlation you don't know which direction the arrow flows so for example I cannot tell if co-pilot makes good Engineers or if good Engineers use co-pilot but I do know that they're correlated so I don't know if I care which way the arrow goes right because if good Engineers want to use co-pilot great if co-pilot makes good Engineers great right if good Engineers ship more PRS great if if more PRS makes good Engineers great like it doesn't matter to me which way the arrow goes but again I'm not trying to be too scientific for us it is an art form to try to think and dissect these types of results and that's how folks should think about it in their company right I want to have my Engineers focus on the thing that matters more for them and so in pair programming and again co-pilot you don't want to look at lines of code because in a lot of times it's less code that you want right and I mentioned that you know maybe I should be ashamed that I have the largest ruby code base in the world instead of actually saying this is something to celebrate I would like less code I would like to subtract more code I would like my simp my system to be more simple it allows me to reason about it better and and as these tools get better you know I want co-pilot to come to me and say delete this module delete this code here's a simpler way of approaching this uh this Solution that's what we want to see and that's why the PA programming Paradigm is actually much a much better uh metaphor I think because the idea is that you are pairing with somebody there's one thing that we did try actually in pair programming you know if folks have done it you know it's two Engineers one machine but two keyboards two mice and they work together is I did try to hack together uh via the accessibility mode this idea of a voice co-pilot so the idea that I could just talk to co-pilot and it would talk back to me while I coded instead of actually doing it by the typing interface I couldn't get it to work exactly but I think that's coming as well where you actually could have like a almost a more natural interface than typing so I'm going to share some survey data and again this is the qualitative data that we had at Shopify about how often folks were using co-pilot what they felt about it um how useful they found it and again you're seeing some pretty high scores for some of the products that have been out in our case like you know we've been using co-pilot for you know almost two years now and when we talk about co-pilot we talk about right I'm talking about the autoc complete of co-pilot right the generalized model autocomplete and again those models are getting better right we heard it was GPT 3. 5 and then G GPT 3.5 turbo we saw that open AI launched GPT 4 Turbo which hopefully is fast enough for for auto compete and we'll start seeing these get even better and as you go down from that you see other some of the other products that GitHub launched some of them announced today that yet don't have product Market fit with Shopify but again we just launched them right shop GitHub CLI GitHub co-pilot for GitHub co-pilot chat co-pilot chat and vs code uh get co-pilot for docs all these things we think will have great adoption as they um find product Market fit inside Shopify again still early days for us but you can start seeing that uh the numbers are actually quite high right 70% useful 75% use it all the time some numbers we do have lots and lots of folks using co-pilot it's funny when I talk to different teams um I feel like let's say it's about 70% of engineering is using copilot I think that number is low I talked to other uh engineering leaders they think that number is high it depends on what your goals are right in my mind we are seeing that there's a slight correlation between using co-pilot and having great impact again don't know which way the arrow goes but it's interesting to see why people don't use co-pilot and I'll and I'll talk about that in a bit we have lots of Engineers who have used it for six months or more I think one of the interesting things there is acceptance rate again I have lots of debates with people is 26% 25% 15% 30% are those High numbers or low numbers so for us I believe those are high numbers what that's saying is almost one in four times we are accepting the suggestion from co-pilot now you could use the you could take the pessimistic View and be like wow so three out of four times you're not taking the suggestion I'm like sure but what happens in the in some cases is you don't take the suggestion but you take the idea oh I see where you're going and then you don't take the suggestion but then you write something similar or you get an idea that it's now prompted you towards in this morning's keynote right we heard um I don't want I don't want to start with the blank page and this is exactly where co-pilot excels because you can write a comment and it can start writing some code for you even if it's wrong it starts you off in the right direction or even by the way in the wrong direction but that is something for you to get started with in an edit and then of course at Shopify Ruby is the number one language we uh we use there and you can see that that's the most uh popular language uh uh that GitHub co-pilot gives us suggestions for but we do see varying rates we do see from 15% to 30% acceptance rate across our repos depending on the language again more survey data qualitative we do see you know things becoming uh as people get more comfortable more and more folks using it over time and again this is Engineers getting more comfortable as more idees get supported by uh by GitHub as well we can start tracking data better and we are starting to see more and more of the senior Engineers also picking up copilot where at the beginning I think they felt it was like a beginner tool so I want to go through again some more qualitative data and just specifically here to show you the difference between what we saw at the mid year and at the end of the year so what's interesting is as people when you know you start asking these questions right we can see that their people's perceptions of what co-pilot can do change so for example people originally felt like I'm going to get more productivity it's going to help me with repetitive code it's going to help me writing tests and over time you can see how they started changing to say actually you know what I'm actually getting productivity boosts it's actually smartly completing my code and so even their perception of using the models and as we all know behind the scenes the models are getting better it's learning but the people are also learning co- pilot is not something that you drop into your organization and it's just going to be something that people adopt and they don't change they're going to change with co-pilot we see this with any new tool right the what's the Martian M on line right first you get the you know the tool you you shape the tools and the tool shapes you that's what happens with co-pilot you start working with co-pilot as your PA programmer and again you start seeing how people change their perception of what they thought they were going to learn with co-pilot with what they actually do and what's interesting here is you start seeing how people are actually thinking through how they want to use co-pilot maybe they're going to start something in a new in a new language that they never used before because they're now feeling like they can get over that barrier right at Shopify for example our our systems programming language is rust and we are seeing people who start using rust more because they're now not afraid because they've got a co-pilot to come along with them again more survey data and you're see again what people are feeling like they are going to be helped with in the midye and where they end up right so people do feel that are appreciating it is a tool that saves time and if you're saving time and again replacing tasks there is a cost benefit to that although it's hard again to quantify right I talk to a lot of again CTO and CFOs who want to quantify the results of a co-pilot or any AI tools and it's tough to do unlike a cost like your salary it's very easy to quantify it's hard to say oh well these people are happier and got to the solution faster even though they produced potentially less code or they were not daunted by starting a new project in a language they didn't know because they had co-pilot helping them so again trying to justify the cost only quantitatively might be super tough and maybe not something you actually want to do because it might take you away from actually having your eyes on the prize which is using it for a while changes the way Developers interact with the tool right one of the things I like to say is it's kind of like surgery you don't want to leave halfway you want to deploy it for I would say a quarter or two before you actually get a feel for whether it's working or not and again in many organizations I just talked to somebody uh this morning who was thinking about co-pilot and how to roll it out in their trial group and whether to look at those metrics and then determine should we roll it out for the whole company and again only looking at quantitative it might be hard because they want to fix figure out well how many lines of code did it write or what was the acceptance rate or how much faster what's our velocity again looking at the Art and Science line it may be hard to see but looking at the qualitative it does tend to show up pretty quickly and of course you know in some of these things you see that you know Co is co-pilot going to generate my boilerplate code for me yeah the the answer is yes and what we find with Engineers is they're more able especially the more senior you get they're more able to spend their time in the more complicated parts of the program the parts that actually are value added versus the boiler play code and the repetitive code or the tests that they feel like might not be where they want to spend most of their time again here's the co here's some data on co-pilot chat beta this is actually uh again things that we're that we're announced this morning we're seeing some interesting data coming through on how our Engineers use the various uh parts of co-pilot to stay in flow you me you know Thomas mentioned this morning you know co-pilot workspace which again is a way to keep you in the flow of your your development cycle so you're not reaching for other tools not going outside of co-pilot not going to chat gbt on the web not going to I mean I don't know people go to stack Overflow anymore not going to those tools because what's happening is you're trying to integrate everything into one single IDE and what's interesting about that is again staying in flow that's what par programming is about that's what copile is trying to do right we're trying to help it's trying to help you improve productivity and again you can see what's what people are thinking about in terms of the things that they want from co-pilot again on the qualitative side you want to make sure that you are thinking through the entire solution and if if you're not actually if you're an engineering organization that is not doing a developer happiness survey or not doing any kind of qualitative analysis you're probably missing out on a plethora of feedback that could be helping you drive your engineering org forward so oh somebody clapped I don't know one clap for that one Okay cool so now co-pilot does have a learning curve what I made that is it is it is not something you drop and then it's be like take it easy everyone we just put this new tool in hope it is working like that's not how I would launch anything I talked you know at the beginning of the presentation really about bringing AI into your workflow across the company and your customers and your employees in a way such that this is one of a suite of tools that your team might actually adopt it's a powerful one and you saw on that you know that tweet I showed where after sales and marketing engineering is one that comes up there but there are things that you have to work differently around with copilot the way you write comments completely changes with co-pilot So for anybody who's used copilot they know that by writing comments is how you prompt the machine right and of course we saw today that you can now use copilot chat to generate boilerplate code as well but what is a main difference is how you write comments and you're not just writing comments again for you you are of course writing them for you or the future you who's going to be forgetting about this code in two weeks but it's also for co-pilot and co-pilot is going to make a guess as to what code you're trying to write and it does change the interaction model you have to get used to the autocomplete because what happens is you're going to have the grade out text and you're going to either accept it or not accept it again what I have seen is even when I don't accept the code it's now priming me with ideas that I might want to use then actually write something else so the acceptance rate may or may not be um an actual indication of how well it's working because potentially it's it's priming me for some other type of uh development that I might want to do it is not perfect co-pilot makes mistakes we saw one in the demo this morning humans make mistakes also your PA programmer makes mistakes you will make mistakes you still have to be a good engineer to work with co-pilot so for the Doomsday or about AI out there saying hey co-pilot or these tools are going to replace Engineers that is not true it's going to make you more effective it is going to replace some of your tasks but it's not going to replace you and the reason for that is because you're going to still have to understand what the code is that's still going to happen and yeah you can you can try to write some things but as soon as you try to modify you want to make sure you are still getting a feel for what is being developed and again this is true of a pair programmer also right like you shouldn't sit with somebody and they write a bunch of code and you don't understand it like you should understand the code that's being written and by the way it will make mistakes just less like a human right now the tools are going to get better and better but it is it is a workflow change and I think what's interesting about it is keeping you in flow is the key now here's an example of one I just literally did this uh did this last week when I was writing this deck where I was like let me just open up co-pilot and actually just write uh a sample code and as you can see I just created an array and right away co-pilot um I actually should have done a better job of like highlighting what co-pilot wrote and what I wrote but co-pilot write wrote a a loop a for loop at the beginning which just iterated through my array from 1 to 100 0 to 100 and instead of using the array length and you can see right away that's just poor code because if I change the length of the array the for Loop will not work as intended but now what's interesting is the comments that I wrote I was prompting co-pilot so I said iterate through the array it wrote the right code it actually used the array length and then I said print the array it wrote the right code it used the array length but the first Loop is wrong right now there's nothing there that's telling me except you know good engineering practices that potentially um I'm it's making a mistake it's not thinking through the problem in a way that's going to be maintainable later right this will probably just work and then someday someone will come in and be like this arrise Too Short change it and now everything will not work so again this a very simple example you can open up your own uh ID and try this but the idea is the models will get better you will get and you will get better at identifying things like this this is a super spicy topic because at Shopify we hire lots and lots of Engineers and I talk to a lot of engineering leaders and they say well my engineering my my interview process now is all screwed because they're just going to use co-pilot and now I can't tell if they're a good engineer or if co-pilot's a good engineer and I was like well if you're if co-pilot can write your do your interview questions maybe your interview questions are dumb because yes co-pilot is going to get better and people are going to get smarter and people are going to become more effective and we ran this trial at Shopify where we I mean right now we allow candidates to use co-pilot we tell them about it in the process and we say you can use co-pilot uh and it doesn't have to be co-pilot it can be any of the you know AI productivity tools because we believe that actually we get to more interesting questions by using these tools right that's one reason the other reason is your interview process should be the almost the same as your job right so if you use co-pilot at work use AI system at work then you want to allow your candidates to use the same tools because then you can tell whether they would be good at this job right it's weird to be like we can't use co-pilot in the interview but then you will use it at work I'm like well you're testing the same things then and so for us we do use it um at work we allow it we don't discount you if you don't want to use it although I'm sure there's going to be some future state where if I ask a candidate are you going to use copile and they say no I'm going to be like why because it's weird like here's a bunch of numbers add them up you want to use your calculator no why like I'm just interested now right now it's not really fair because they might be coming from a company that they don't have co-pilot and you're like I don't want you to learn how to use co-pilot in the interview so it's fair but over time it's going to be weird that you don't use an AI assistant in your coding interview and and one interesting anecdote is I do a lot of interview shadowing I talk to lots of uh engineers and I Shadow the interviewer because I want to evaluate the interviewer and I will turn off my camera and mic and just watch candidates and whenever there's a candidate who's not using co-pilot what I've noticed because I tend to try to do the same questions as the candidates to kind of see if the questions are you know ones I think would help evaluate uh an engineer correctly is every time I use I have a candidate that's not using copilot and I am I tend to finish the question twice as fast so if a candidate has 90 minutes to do a question I tend to finish it in 45 minutes I leave the call send my solution to the interviewer to be marked just like the candidate but what's funny is an old you know Rusty programmer like me can actually quickly get up to speed finish these problems very uh in a way that is very very fast so just some ideas around legal this is the number one thing I hear from engineering leaders how should I think about the legal aspect of code being generated by open source repos by co-pilot now here are two press releases I didn't get a chance to put the third one in which is uh open AI has a thing called uh copyright Shield right which is the idea that they're going to help indemnify you one of those is from Microsoft one of those posts one's from Google and basically what they're saying is uh we will indemnify you if you get sued for using our tools that's interesting I don't think that's the thing you want to rely upon I think the real answer is to work with your legal team early and often to Think Through how you want to deploy these types of tools in your company so imagine um your designers want to use mid-journey same idea you should be talking to your team about how to use these things versus just saying at the end of The process by the way we're going to use this tool help me get is through procurement legal like that's not how to work with your team all right so what have we seen so far at Shopify I mentioned 70% of the engineers we are and about 26% of the code completion we are almost at 1 million lines of code written by co-pilot pretty amazing um still a small amount of code in the Shopify sense but still amazing that that uh co-pilot is being that productive for us that we have this amount of code in our repos already and you can see there are almost 700,000 acceptances by Engineers of the co-pilot suggestion right and the reason that those numbers are different is because a co-pilot suggestion is one thing but it might generate like 10 lines of code and so that's not even including all the prompting that you get that might make you land on a in a in a direction that is fruitful but you don't accept the suggestion in which case it won't even show up in these stats right it won't show up in the lines of code it won't show up in the acceptance rate so it's probably higher than that cool and then of course happier Engineers right which is something we should all strive for all right and then lastly it's not all roses there are some things that co-pilot um we hear from people who don't use co-pilot right it gets in the way feels like I'm cheating which is an interesting one right we hear that from candidates I don't trust it right so we do see some of it around to slow that will get fixed or the suggestions are not that good these will get better over time but again just for your organizational um you know perview you can start thinking through why an engineer may or may not want to use co-pilot in summary reach for AI early and often and if you're using GitHub it's a no-brainer to use Coop palet thank you very [Applause] [Music] much

Channel: GitHub

Views: 10,845

Keywords: github, git, git and github, github tutorial, how to use github, github for beginners, code, coding, programming, developer, software, software development

Id: wVKBwcm5dbw

Length: 40min 23sec (2423 seconds)

Published: Tue Nov 21 2023