GitHub Copilot and AI for Developers: Potential and Pitfalls with Scott Hanselman & Mark Downie

[Music] hey friends how are you all right thank you so much for sticking around in the evening and having so much fun with us today at Microsoft at night I am Scott Hanselman and my name is Mark Downey and Mark and I are going to talk about all the stuff that you've been seeing today and try to put it into a little bit of context because I think that there's everything is a co-pilot right uh and now you're going to have to go home and you're going to have to explain to non-technical Mom and Dad what the heck's going on I've had to have my mom call me and say do I want the AI is the AI in the room with me now is the AI going to hurt me she hears about the actor strike in AI she hears about she wants to know if it's a virus does she need to remove the AI then Alexa's saying stuff why is Alexa not smarter a lot of questions happening around AI people are saying that it's going to change everything but we're not really sure how it's going to change everything and the important thing to understand is that we're just getting started this is a new user interface we're spending time figuring out where buttons go on the screen but we also need to figure out how do you talk to your computer it's not a person should you talk to your computer should you treat it like a human should you be kind to it should we name them these are all all things we need to figure out that's the interface with our computer when iPhones came out and touch screens came out we had to relearn how to do this we're in the process as a community of relearning how these co-pilots are going to work aren't we all right so what I'm going to do is I'm going to go into my machine here and pop into the open AI playground I want to point a couple things out here this system message inside of the open AI playground and for non-technical people in the room this is where you would spend time if you were going to start to train a chatbot to talk about a subject it might talk about your Museum or your topic of interest if you are an author and you've written 10 books you could feed your books into the assistant and then interview it and it would now become an assistant that knew everything about your topic this particular assistant in this tool here in the open AI playground only knows that it is a helpful assistant but you don't see that as the user of the thing you are just assuming that the agent is going to be helpful right so if I go to this assistant and I ask it something like I need a taco recipe we have to ask ourselves how does it know how does it know anything about food how do I know that the recipe is not going to kill me like we have questions there's no kitchen to test that recipe how does it know what the next thing to do is if I drop into this complete section here in the playground I'm going to go down in the lower right corner and there's a section that's marked show probabilities and this really showcases is that when you talk to an AI you're not talking to an intelligence you're talking to something that picks the next likely word now yes this is an oversimplification but that's what analogies do now I've been married for 25 years and you just had your 25th anniversary certain not married to each other just a coincidence 25 years yes 25 and does your wife know what you're usually about to say most of the time okay you surprise her sometimes rarely rarely right because you've known each other so long for a quarter of a c a quarter of a century you finish each other's sandwiches sandwiches exactly so if we were to go to this this this uh artificial intelligence here and turn on the full spectrum of possibilities that we're going to actually see something and I'm going to say it's a beautiful day let let's go to the what is the next thing to say what's your wife going to say the next word is we went to the mum the museum right so this is effectively AI as Family Feud right Steve Harvey's here it's a beautiful day let's go to the show me beach show me Park what's the right answer let's find out Park this is where I need the non-technical people the business people the business development people to understand is that answer right is Steve Harvey's answer right if it's the first answer is it right no it's most likely to be the next answer that doesn't even mean it's the most likely to be the right answer it's just the next one let's click on this because we turned on full spectrum and look's look at that so now if I say as a user of AI as a data scientist as a prompt engineer which I've now added to my LinkedIn I I'm sorry Mark you're wrong museum is wrong it's what we did it's what we did it's what we do because it's right for you right so then why is Museum not there at all what are we missing some context we're missing some context because we're missing 25 years of relationships of time spent together of the things that you do right I'm here with my friend Mark he's from London period his wife and he love going to the museums and they love art oops okay I'll spell that correctly and let's see if that context provides any value here whoa right yo they they love you the AI Loves You this is amazing look at that Modern Art Is this a real thing it is Tate is is a real Modern Art Museum it's a British thing yes Yes actually haven't been to that one you haven't been so the hang on did the AI just provide value to you and you're like we have not gone 25 years never been to the Tate Art Museum isn't that brilliant look at these different colors it's saying it might have suggested other museums y once it said Tate modern some suddenly became more right uh more reasonable and then art is read it was considering punctuation I mean what's going on here Mark so my assumption here is that every single word you getting imagine you're just sending somebody a text right and it starts to autocomplete for you that next word may be useful based on what it's seen from everyone else but for you it may not actually be the right thing that's why we end up sending each other those terrible texts that make no sense because the percentages didn't work for your situation and so this is every time you see some red we starting to see something that's less likely that's happening or more likely as the case may be so you've got to think about this as every single thing you context you provide kind of gives you a percentage on what it might be in the next next word exactly so if we return back to the recipe Theory remembering that the only context it has is you are helpful right then what it's going to do is it's going to look around What's called the Corpus the Corpus is the thing that you trained it on and in this case the open AI very large language model is trained on all the text on the internet and this is really important because the internet is 49% a joy and 49% pure evil and 2% that haven't decided which side they're on yet and what will happen is if you are cruel if you are unkind if you are mean if you are impatient if you say mean things to the AI the next word will be mean cruel and unkind this is not a joke and it is still not a person but if you say nice things if you complement the a you're going to end up pulling from the nice parts of stack Overflow and Google and Bing and the places where people are Pleasant so I need a taco recipe right now and you're helpful are going to bring me a delicious and easy taco recipe okay it's picking words that are near Taco on the internet but if I said something like you're a belligerent assistant a lot like um Benedict Cumberbatch as Sherlock Holmes period you're not super happy to be here comma and you're going to give me what I need but you're not going to be thrilled about it period and you're a Little Bit Sassy okay so now I've given it context now remember you the audience cannot see that you've just showed up on Scott's website and asked for tacos oh wonderful another banal request for a recipe sorry I apologize on behalf of England KN yourself up yeah yeah oh taco showells sof or hard it won't hinder oh this is just mean it's getting it's very quite cruel here look at this your lack of preference won't hinder my guidance here's a recipe I can't promise it'll be stimulating as a true challenge for the intellect it's a rather simple and mundane task suddenly it's rude it will be a different flavor of rude depending on the prologue the point being look at that how oddly common we don't see that I programmed that I told it to be unkind when we interact with people a we always say the first impression is important someone's like you know I met this guy at the at the hotel lobby yesterday didn't really like him not really feeling hlan not a nice guy maybe I had a bad day maybe I didn't have U you know the three Diet Cokes that I need to wake up in the morning there's lots of reasons why I might have a bad first impression presumably after 25 years of marriage having each been renewed for another season right uh we have made a good enough impression that our wives tolerate us right for one more season yes yeah for at least one more season this is important because our wives have that context and that new person is still building context and context is so fundamental when you are interacting with a co-pilot whether it be a Microsoft Office one or a visual Studio One now we've got some developers here we've got some non-technical people here we've got some it folks here but I know we like to see secret things from inside Microsoft right we like to see stuff that's never been seen before correct you have some hot bits this this stuff's new now this means that this is Fresh Off The developers computers right this is not released is it a is it an experiment is it a preview where is this so this is going to be a preview soon previews coming out really soon so the bits I have will be coming out really really soon okay so we're going to show you something that does not currently exist I'm going to let Mark take over and we're going to see some things where we have co-pilot GitHub co-pilot plugged into Visual Studio for the non-technical people in the audience of course Visual Studio is like Microsoft Word for programmers it's the place where we write the stuff and then when we ship it if you write a script you ship it into word and if you write a program you ship it from Visual Studio but it's just text right it's just text yep yep so I'm actually going to try out asking co-pilot to help me with a prog set of programming problems um actually I'm an open source developer I've been doing open source for many years now in fact I took over a project that Scott used to um to used to Shepherd quite a long time ago what I'm going to do with this is see if I can get it to fix a few of the issues that we've had open for a while in fact what I have open right now is the issu section of GitHub where all our open source code resides these are bugs right right these are bugs that or enhancements that folks want as part of the the blogging engine they want it to be a little bit better and um what we tried to do here is create a scenario where if I create a blog post and I add an image into that blog post I want that to be the hero image now naively I this was originally implemented to just pick the first image but I want it to be explicit I want to say that image is the one I want for my blog post let me show you what I mean by that so this is my blog post here my blog right here I'm going to pick a a blog post and just so happens that this image is the only image so it will pick this one but if there were three images it would just pick the first one anyway and it could be a gift silly gift and that's not what I want as the hero image right okay so I write a blog I've got three or four pictures and you're going to pick the best one and it'll become the front the hero yeah but you're going to do that programmatically and this doesn't exist in do blog does not exist today okay in fact this has been open for two years you can see no one wanted it no one touched it things are usually fixed relatively quickly but no one wanted it I think it's because it's a regular expression problem and and just as a point uh to to to Mark's credit this is a blogging engine for writing a Blog it's called dos blog which is German for the blog uh this was actually written by Clemens vasters and then myself and then Marcus taken over as a shepherd and it's all open source so all the code that we're showing you you can see it's completely free and open all right so this is the first method this is a a function in a ex in a extension class and this is what where how we grab that first image right so as you start there a kind of new contributor you may ask yourself well what what is this method doing and I kind of like the idea of asking in a conversational way using co-pilot to ask and I hit I'm going to hit alt slash and that will bring up a little inline prompt and I can simply do actually a slash command and if I tap down I want to zoom in on that yeah yeah so I want to point out that he's writing his code right just like when you're writing a Word document you have a comment that appears in line he's made a little inline Mark there he can go and ask it questions and slash explain you're basically saying explain but how does it know what to explain so so so it's sitting in the context right now so the thing we were establishing right at the start here is that without context it's just going to do anything it wants the probabilities are going to be and the completely uncertain what we want to do is establish context if you don't establish context you essentially get more random and more random um suggestions so let me go ahead and hit enter and see what it comes up with right and this and this context is a kind of ground you're grounding it in the thing you're trying to get it focused remember that we see folks taking chat GPT and open Ai and writing limericks and do this in the form of Shakespeare none of those things are interesting he is saying tell me what's going on in this code it still has the power to do limericks and Shakespeare but that's not what we're doing he's grounding it in a reality which is explain what's going on in this code so this already knows I'm a c developer like I don't have to set that context anymore and it's it's given me a pretty good answer here it said this is a method that's in a string extension class it's told me the name of it and what the information it's the parameter it's using it's basically saying there's a regular expression in here so it's read all of that detail about my code as he's able to share it with me so one of the things I used to do when I was a developer at first was literally write out the explanation of the method this kind of sits within that kind of practice right of learning where you can get a natural language explanation of the scen real absolutely so much time is being spent focused on the what AI can generate but not enough time is being spent on what it can do to summarize the tldr that too long didn't read right that was effectively a technical writer just wrote you a chapter of a book about a function that might be unfamiliar exactly that's what is going to change for the next generation of developers yeah open source developers now don't have to kind of spend those weeks gearing up you can cut that time dramatically by simply asking um so I'm going to actually go a little bit further because it's a regular expression and I don't know about you but every single time I do a regular expression it seems like the first time I've ever done a regular expression so let's kind of see if we can break out what this regular expression is doing I'm going to hit Alt Enter alt SL G right the regular expression for the non-technical in the room on line 79 that looks like a swear word like maybe it would be in a comic book with Snoopy and just he's very upset with uh with the little bird that's sitting on his thing those are we have to write those like like bespoke and they're the worst right so let's go ahead and it hit enter see what it comes up with so I to to situate make sure it was in the right context I put that on the right on that line right that line I'm I'm telling you this is the regular expression by highlighting the line I'm interested in okay I love this so what's it shown here it's shown it's looking for the here's the breakdown this kind of does a breakdown a regular expression I'm looking for anything that starts with the image it's looking for anything after that and it's looking for something called SRC which we know to be source of the image right it's breaking down each part of that regular expression I think even with this information I could probably write the regular expression again um but we're actually going to go a bit further than that I think absolutely beautiful it's in plain English it is so powerful to be explaining in plain it's an infinitely patient person spending time with you unlike if I call Mark in the middle of the night on a Sunday and I go Mark help me understand this regular expression ah right handlan stop calling me this is not programmed to do that it will be infinitely patient yeah we have grounded the responses here so you don't have to worry about that part right ultimately we want to make sure that it is a helpful assistant and not Benedict kbat so that we love you're watching Sir okay let's let's just jump back out of here um so actually so so what I want to do then I want to do find first image but I want to do find hero image in my example right so I'm going to go ahead and this is what I would normally do I would start kind of start the kind of outline of this I would change the name beginning to write a new method right yes sir hero miss it hero image and here I can again start asking questions with alt slash again I'm going to say rewrite this method using a while he's doing that in coding we are the bridge between the business person I want the thing done which business people usually speak English or uh business people English uh and they write it up in a Word document and they email it to you and then they go golf and then we the developers are left to actually translate between English Pros Business Solution and turn it into some kind of code what's amazing about somebody like co-pilot is that in this case here he is writing Pros very specific Pros he's giving a very specific instruction and hoping that that will translate that specificity will translate to another kind of specificity right so I'm said just to just to kind of summarize rewrite this method using a regular expression that checks the blog content parameter you can see right there on line 94 and Returns the value of the sour string that's what we want only where the class name is set to Hero image we only want we want it be very specific not the first one so I'm going to go ahead and have that go now probabilities apply right I don't know it might say Museum you don't know we just don't know okay how do you know if it's right though because we just talked about how museum is right for you and and you know Beach is right for me we we have to test it ultimately I don't trust anything this says I mean to be respectfully truthful here I have to assume that it just it's like um when I go to a blog post I go to stack Overflow it may be right for them but but it may not be right for me I'm just trying to like I want to make sure it's a little subtle just want to make sure we're clear AI suggestions might be inaccurate why AI can't make tacos it made a recipe right we have no proof it's good tacos you have to make the recipe AI can't code yet and in this case here we have to test it right so I'm going to go ahead and accept that and we'll do a quick we do a very I expect you'll do a thorough review but I'm going to do a quick one here standing here oh I'm already seeing that it is kind of put my um uh tabs not quite right but let's compare it really quickly to find first image so find first image is here and it has a variable find first image it has a naming convention it's doing its thing find hero image looks almost identical right don't know if it's right yet but it looks almost it looks a little different there you see hero image yeah yeah you see Class Hero image it understood the regular expression in fine first image and it in fact um renamed the variables to look like hero image so my code is sitting in this context right and it's been changed already just like you would do if you went to stack Overflow you'd see some code and you say Ah that's almost right right let me pull it into the look more like M did it run it did it compile it did it test it no that's all my job now that's your job let's in fact let's go do this let's uh let's do this I'm going to go into profile post and here I'm going to change find to find hero image there and let's hit the start debugging so he hits to start debugging for the non-technical it's a little VCR button at the top with the little play he's going to run this application with code that was just generated you really have typed nothing but pros at this point I have not done anything so how will we know if it works we should see our image here which we don't okay and that makes sense why because we haven't defined in whatever the source of this is as class equals hero image let's let's go back in really quickly let's do that yes sir uh so we want and remember we all XML files here so we've got class equals uh so he's got a Blog just like WordPress or your own websites or your blog in this case here it's using dos blog and in order to make this demo work he's going to go and do this manually but you would not ordinarily have to do that manually but if the class is correct we should see the image appear which would mean that you didn't have to write any code and there it is it picked up the idea that this class equals hero image it's there it did it okay but we talk about context we talk about how the AI doesn't know 25 years of marriage it doesn't know what I ate for breakfast right it knows only what you tell it and that context those references like if some kid turns in a five paragraph essay I want to see references I want to know where you look those things up right let's see some things that no one's seen before yeah okay let's let's move on so you probably would have seen AI write code AI read code actually like to move on to how it might help you debugging and Diagnostics I this this my key area here um so we've been implementing with DOs Blog the idea of activity Pub standard which is a way hopefully that we will allow blogs to kind of communicate within the fediverse so this standard is the way we do that so I've been may I translate please do uh so Twitter is awful and there's another place uh in the fed the fetty verse the Federated verse also called Mastadon is an alternative social network there's lots of social networks and just like you've got threads and Facebook and Instagram and all these different places some of which have varying levels of awful what you want to be able to do is you want to take the information on your website and Syndicate it just like Garfield writes a comic and it shows up in local in local newspapers he's going to write his blog post and he wants it to show up in the feeds of people on mastadon and Twitter and all the different places the collective Federated universe or fediverse right thank you sir so I'm going to dive in here with some Diagnostics here and what that really means is I want to test my code I want to see if this new code that I've been creating is going to be helpful and the way I can do that is by in fact I've got a uh HTTP file which is a basically a way for me to send um a request to a particular endpoint so I can test bits of my blog to see if it meets the standard would you mind if we just dug just into that just a smidge deeper C can we look inside there yeah sure thing so this is a HTTP file um again this kind of I this is kind of the variables here but this is essentially me sending a get let me zoom in a zoom in on that so I'm sending a get to this URI and it's you can see it's taking the variables here uh for the endpoint and this is the header the accept header is part of the header of that request so he's effectively simulating being a browser simulating being like Postman and he's basically doing this locally on his computer and because his computer is not accessible from the internet he's going to go and make a developer tunnel allow that to be available to the outside those HTTP files are a way of doing kind of live testing right exactly okay so I'm going to go ahead and send that request and what you're seeing coming up right here is called the exception helper okay um this is the exception helper is my way or visual Studio's way of saying something unexpected has happened and if you don't handle this correctly it's going to mess with the the part of the the code that's running this is all brand new to me right this is a um most of the code in here is dealing with creating files from objects and the vice versa serialization deserialization what we know is that and so this is a particular kind of error that I'm now going to have to track down it's my responsibility to track that down right and in the movies and TV there's a lot of Glamour with a person slapping the keyboard and they're like ah I can see it it's happening and then another person sits next to them and there's two people on One keyboard most of programming is doing this oh yo what do I do now and then you stare at this for some number of hours right this is broken we are crashed we are stuck on line 28 and it's 2: a.m. and our homework is due in the morning right yeah I'm at the top of the funnel anything could be going wrong right now um it could be it's stopped here but it could have been anywhere else and hopefully what we I'm going to show you today is some of the newest Parts we've got got thinking about how we can help you kind of shorten the possibilities of what might be going wrong um with the code all right so I'm going to go ahead and ask co-pilot there's a brand new ask co-pilot button we have right there I'm going to ask co-pilot now this is starting a conversation rather than just be in line I've got a conversation happening here um and this conversation um is being done for me I asked co-pilot and it's taken some context from my debugging and handed it and said this is going to be helpful for exceptions and this is this is huge because a lot of us and our interactions with chat CPT when we see demos and everything is live none of this is canned to be clear we don't do can demos is you ask the thing a question and it gives you an answer and then you Marvel how smart the AI is this is not that the AI is not smarter than us it's our helper it's co-pilot not pilot pilot it it really doesn't know what it's going to do we are going to be the ones who have to be the direction for this right but what is going on there all right so inside there let me go back in again this is the the references the context it has passed to co-pilot in order to the Exception by itself you can pretty much look that up they we got docks on that but in this situation to understand what it really means you're going to need things like the call stack where did this happen you know what what happened before this um you need information about the exception and you probably need to know something about the local variables in that particular situation so we are being upfront this is what we're sending when we when we have an exception we want to help you analyze so when I asked chbt earlier about it's a beautiful day and where should we go I gave it context I I my on my own I decided to say where you were from how long you've been married and what you enjoyed doing I suppose it could have said I don't know enough about you to say where we should go today right so in this case here we gave it some stuff as context is it smart enough to actually ask for more this is what we're working on re most recently I want to show you here okay so it's gone through and it's actually got a good description of got a pretty good description of what's going on here but that's we actually really got that from the exception itself what's interesting is that it reckons that it's something to do with a known type there may be a few of you who know what that means here um I I do not and so what he's asking you for here is if you give me just a bit more information and he said could you please provide the definition of user page view model right it knows about this because of the narrow context we've given it it's saying that thing I need a bit more information on I need more context Why didn't it just take it because it's all powerful and it sees everything and it knows everything Why didn't it just take it and that's the thing we've got to have this balance between what we are going to give you and what you actually want to send right and I think it's important for us to kind of be responsible at this moment to think about the things that you want to share or don't want to share ultimately that's a decision that again why this has to be a co-pilot you're going to make that decision you're going to say what things do I want to send to co-pilot this particular file he full of secrets don't want that ever going there's no good reason for that to go you're going to be the one to kind of be in charge of deciding when it's going to be something else this is so important what Mark is saying that we are make we are being intentional and by we I don't mean Microsoft I mean we in this moment in an AI moment have to decide you know those movies where Arnold Schwarzenegger wakes up and he's on mars or whatever but he's not on Mars he wakes up and he goes over to the computer and it's how do you know it's the future cuz he has a giant TV that's mounted it goes hello welcome I analyzed your urine this morning and your sugar's a little elevated and I'm going to put in a call to your doctor and by the way your fridge doesn't have enough milk I've ordered some milk for you and we go oh the future's going to be great that's super creepy and we don't want that right if you sit down at your computer we have the technology we could write a program that looked at the webcam and said you're looking a little peed today right you should I you should smile more Mark and you would say wow that's now entering uncomfortable right so this user interface experience that you're developing with co-pilots is not about the tech it's about the humans it's about the decisions it's about the responsibility and the intentionality about what we're going to do so you at the co-pilot team are deciding what should a pair programmer know and how should it behave right it could have taken it but you chose not to let it take it exactly so I'm going to go ahead and continue with this um we're in uh control four mode there okay there we go all right so it's asked me says could you get bit more information sure you can so I'm going to click on this it's going to bring up this class and if I can remember shift alt equal sign yeah so what he's going to do is he's going to do an expand selection so what the AI has done is it's asked for more information so he went to that place where the information is he's expanding the selection contextually and then look at this I want to just zoom in on that for a second if you wouldn't mind I want to get that to pop up if I may yeah please and then I want to show you he he can look look at that he can say here are the things that I want to share but look at look at the one there that say selection I'm only going to give you from this line to that line and I want you to look at that and I'm going to say this is the thing you asked for all right let's go that so let's do that again live say here's the selection from these lines let's go and then we see that reference is again showing the additional new information and now it has more ideas for us wow it doesn't have more ideas it literally made a whole suggestion and said go and write it like this yeah here you can hit control four come back out sir all right so here it's literally telling us hey we are missing this known type information from this user page view model this is the reason why um this broke in that way um I think I'm going to have an issue with my preview there so I can copy and paste that into place but ultimately that's the the problem right but it wouldn't have been able to get there without that additional context so it's asking a question it's the AI realizing hey we actually we've got a theory but we need to validate that and the only way we can validate that is if we bring in more information this conversational aspect is the very newest and latest parts that we've been working on here and um we've got a couple I got one more example if we have some time yeah absolutely and the thing to point out here is that we want AI as I've said before publicly to be more like Iron Man and L like Ultron or Iron Man 3 for that matter cuz Tony Stark was like flying around but he was just chilling and the Iron Man suits had nobody in them Iron Man suits got to have people in them doing cool stuff otherwise they're just scary robots I want a co-pilot that is going to have a conversation with me it's going to be patient it's going to be kind it's going to be helpful and it's going to be that thing that I can turn to that I can ask for advice it's almost like having an infinite technical book you could sit down with co-pilot and say I want to learn HTML today and have a whole conversation it'll walk you through it like a tutorial and that's exactly what we expect from our systems I don't want to say write me a website from scratch it'll be fine and then let it have the keys to the kingdom right okay so what have we done sir so um I want to run quickly just move on to the next area yes sir because I think uh this conversational aspect is really an important way this is the way you you actually debug issues um you'll ask ask somebody you'll ask that one person who's the smartest person in the room and they'll get you the details you want and maybe the answer they they have is informed by what they were working on um however your context is more important so you don't just accept the thing they say you have a conversation and then you add a little bit more context and they say oh soon as you said that let's do this right and so I want to kind of show how we're experimenting that without a game so I'm going to run test Explorer again this code isn't very reliable so I'm going to go ahead and and zoom in here I'm going to debug this particular code it's it's a part of my test suite and I'm going to see why it's crashing so then you hit control 4 yeah got it okay so it's really just started running this test and it's going to go ahead and it stopped again I have another serialization issue this is another serialization issue I've I've created here um and I'm going to go ahead and ask co-pilot I want to notice on the right side however that it started a brand new uh discussion like this is an actual multiple discussions if I can show here by zooming in again I've now have three discussions that have been going on um the original thread that I came in with my pre previous um exception and now this new exception and it's important to understand that each one of those is a separate conversation because I'm building on myself and I'm providing more context if I post to dive into my previous conversation with this new context it's going to basically ruin the original and the new conversation that's a really great point cuz you're effectively got three support tickets right opened with tech support and by giving them that context and saying oh no no this is about that other issue that I'm dealing with I don't I ground it in the focus of the now yes AB because we all are juggling things whether it be Microsoft 365 in a Word document or some email or in this case some code we want to ground it in the work that I'm currently doing so um this analysis again references we've sent the same TP typical same kind of thing we are thinking about the call Stacks call stack is the list of methods I've called up to this point the exceptions and the locals and it's looked at this and again it's asking me for more information right it's looked back at the call stack and said can you provide me a bit information there's a there's a there's a a local variable that we don't have information because it's further up the stock this is really important and this is really significant because it's saying can you check the value it didn't even say give me the value check the value because we don't want to send someone's Social Security number to co-pilot we don't want pii or personally identifiable information so it's saying I need to know if this is is good or not if it's empty or not it's got questions it's up to us to decide if we're going to send it to it and even if it was sent we wouldn't do anything with it I like that it didn't just know right absolutely so I'm going to go back up the call back further back whatever method called this one to see if I can get information here and that serialized variable is right here and I can simply hover over it and see that it's empty so I'm going to say back to it the way it's said to me it's uh sorry you have an extra little serialized serialized the uh serialized uh variable excuse me is empty okay empty all right so I'm letting it know the information it's missing that's look at that look at look at he like that's the issue that's very confident a little sassy there that's the issue I wish they would have been like nailed it you need to ensure it's not empty before you realize okay this I'm going to lean into here because this actually is incorrect I would have taken this information and gone off on a direction that is actually Incorrect and I think we have to kind of as we kind of all getting excited about what it is to use Ai and what it is to um um make it one of the tools we we assume we can we can rely on I think it's important to realize that it could I mean we've pointed all the arrows at it that it could be wrong right and here it's done the 22% thing rather than the 88% thing right the 78% thing uh here we've got a situation where in fact the the scenario isn't going to work it's picked up on something and it thinks it's right how do you know it looks right exactly so as you run through this scenario you'll end inevitably find that this scenario is wrong and you've got to test it you've got to accept what it's doing go through this the process check to validate it put tests around it and then you're going to get to realize that it's another tool and just like you go to that blog post of that one person that's excellent and sometimes it works for you and sometimes it doesn't you should simply ask the question again like ask the question again whose fault is it that it didn't say it's a beautiful day let's go to the museum probability and non-determinism it's just it's it's ultimately the the the issue here is that it's never going to give you necessarily the same response twice and you've got to follow what you know logically about the code it's going to be something that you help with but not something you end with ultimately I'm glad that it didn't work because it's important for these kind of things for you to understand that just because it works in a demo doesn't mean it's always going to work for you we saw before that like beach might be 70% in this case you said this is the 22% you just made that number up yeah yeah but in this case it's this is problematic by the way it's confidently wrong right so and that's part of the grounding and you know there's a there's a balance we have to get to where we assume that you want to have something that sounds relatively confident we can make your assistants do that um we can make your assistant seem I'm this is an answer but I'm really not sure if it's even correct okay right the question is what is the balance so how do we recover from this now let's because you're saying as a as a as a professional programmer of many years you know that that's wrong can you tell it that that's wrong and have it try again so you can provide feedback on any response you get that's something that is incredibly helpful um as we as we kind of um move move in that way um but ultimately like I said when you're at the top of the fal of some kind of Diagnostics you hope to eliminate a few few things and go down a path you check if the path is good if it's not you go back to the top and check some more things so I'm going to basically go back to the top of the path and check some more things again that's great so let's go ahead and I've been having this conversation with my son as he's been applying to colleges and he keeps telling me if he wants to run if he's allowed to run his college essay through chat GPT and I keep trying to remind that chat GPT has never gone to college before you are the one going to college right we need to make sure that we are able to decide for ourselves what's right and what's wrong and having that critical thinking scale is super important this is a a tool it's a spanner uh we need to decide what we're going to do with it okay so you've returned back to the uh uh the exception there and you've said ask again yep okay so you're looking through your reference material you've reased it and are you hoping it's going to get it right again yes I'm going to I'm going to I'm I'm hoping again that the first response is going to be correct okay um again please check the serialized method that's exactly so then tell it it's empty okay serial seriz G to get you turned on to dictation I don't I don't type anymore man I'm always talking to the thing it is empty empting all right let's see if that yes sir ah okay H okay so it's asking me now calized is empty it's asking me specifically about the other method that is further up the call stack it's saying two Json might be an issue can I have a look at what so it had a different idea to be clear you asked it the same thing you typed serializes empty it said nondeterministic yeah it is not 100% Going to say the same thing so I'm going to to jump across here really quickly and here is the two Json I'm going to go ahead and give it that selection so you're telling it about that chunk of code and you're saying hey this is the thing it has an opinion it has something to say h interesting so it's basically saying um we did something wrong in 2 Json it's looked at two Json and said hey um you're not resetting the stream it's one line it's a weird one um and I'm going to preview that uh you think you could have found that on your own or is just going to this is a Time absolutely would not have found that on my own okay I absolutely would not have found that this is what this is a head banging problem absolutely so I'm going to go ahead and accept that and pull that in he even wrote a little comment for you yep reset the uh stream here and then our remaining minute we'll see if we can stick The Landing actually see test Explorer no pressure brother run run go go if it turns green yo nailed it yeah yeah yeah this is the yeah this is the coder equivalent of like he needs this for a birdie oh yes good job yeah thank you yes for the non-technical in room they're like I don't know the Red Dot is now a Green Dot but that is a combination of the co-pilot which helped us and the professional developer who knows better making it happen I want to thank you all for spending time with us we've got a bunch of exciting stuff happening at knite I want to encourage you of course to sign up for these features if you want to get this co-pilot chat preview you can take a look at these URLs or take a picture of that URL right there certainly you can go out and Google with Bing and learn more we appreciate you all and thank you so much for spending time with us [Applause] [Music] today

Channel: Microsoft Visual Studio

Views: 7,633

Keywords:

Id: ZWLsZsa8dgI

Length: 45min 1sec (2701 seconds)

Published: Wed Dec 27 2023