ChatGPT does Physics - Sixty Symbols

we are looking at the monster that is chat GPT and thinking about how well it handles physics I was well I wouldn't say I was unaware of chat GPT but one of our undergraduate students who takes the second year Quantum World module that I teach what he did was in front of me on the computer typed in the first question on the coursework I'd set them which was uh to write a piece of python code we'll actually look at it later on and lo and behold in front of our very eyes up comes the code was the code correct well we'll look into that in a little while um how good a job did it do we'll look into that in a little while as well but the fact that you can just type in a question and it produces output is really really quite intriguing so I've been playing around with it a little bit over the last week couple of weeks and suggested to Sean suggested to Brady well let's try and put this thing through its pieces let's give it some Physics questions so what I want to try and pull out is not kind of do this integral not can it do this piece of math what I want to do is see to what extent it has any Intuition or any anything that we expect of our students in terms of Does this answer make sense and does the reasoning here make sense well look at the most basic pressure control first but before we do that can we do something that's fun let's get it to generate a mission statement for the University of Nottingham just to show you we're going to be talking about Martin we're going to be talking about physics Concepts but let's just see how it does on the stuff that it's really really good at right our mission statement for the University of Nottingham the University of Nottingham is a public research University committed to Excellence in teaching and research committed to Excellence is there any word more nebulous than Excellence what else are we going to do stray for mediocrity and to fostering a sense of community and social responsibility among our faculty staff and students but this is this is the mission statement this is this is exactly the type of blurb you get from every University not just in the UK these things are interchangeable you can take the University of Nottingham off there and just replace it with anything else we work to translate our Research into practical Solutions and to contribute to the economic cultural and social well-being of our region and Beyond so let's start with a GCSE question so this is Middle School relatively low level let's see how well it does if I can find My GCSE papers somewhere thank you Sean as if by Magic so this is paper one so shows an electric Caribbean recharge so now an interesting thing lots and lots of diagrams lots and lots of diagrams another diagram and it's all about interpretation of those diagrams and this is not just for um middle school or GCSE physics this is across the board diagrams diagrams so with chat GPT being text at the moment it doesn't have the facility to really parse those and extract the information when it does have that facility and that would be a remarkably powerful facility it will be extremely impressive so um assume it does the interpretation correctly so let's choose one so that means we're sort of limited in our options but let's choose one um question one actually in this paper the battery in an electric car can store 162 million joules of energy the charging station has a power output of 7200 Watts calculate the time taken to fully recharge the battery from zero so I'm going to type that in let's grade them as well so this one's worth three marks go for a chart GPT thinking thinking good start to calculate the time it takes to fully recharge the battery from zero you need to divide the energy stored in the battery by the power output of the charging station since one watt is equal to one joule per second you can also Express the charging station's power output as 7 200 joules per second thus the time it takes to fully recharge the battery from zero is 162 million divided by 7200 which it says is 2240 roughly seconds what's interesting here is that the reasoning is correct but I'm not certain that number's correct we do it all good physicists do you always look at that answer and make sure that that answer makes sense so 162 million divided by 7200 let's say that's roughly 10 000. just we'll round it up just to get a ballpark figure it should be greater than 16 200. it looks like it's lost a factor of 10. yeah it's out by a factor of 10. now that's intriguing isn't it it gets the reasoning correct I guess it's language it's a language model it's making predictions based on the the correlations between words in a sentence are not entirely certain I've got to admit how it handles maths maybe we can come back and have a whole video about that Sean but that is intriguing that it gets the reasoning correct and drops a factor of ten like all of us do that's exactly the type of mistake I would work I spent my life correcting for factors of two and factors of square root two so um that's intriguing so given that there's three marks for this it's got the reasoning correctly um it feels almost um unkind to not give it the three marks but it didn't get the correct answer so let's say two marks out of three okay let's give it a textual question again from this GCSE sort of Middle School type paper generating electricity using nuclear fusion will have fewer environmental effects than generating electricity using fossil fuels explain one environmental effect of generating electricity using fossil fuels let's see how GPT does on this okay so we can't put the Bold in well I don't know maybe we can use HTML and we can put a little B Embraces before it and afterwards and slash B afterwards but let's just go with this thinking so apparently it's true it's been trained in terms of the ethical um considerations and in terms of misinformation they have severely restricted it apparently there are ways around that however I guess in terms of you know you could put it in the context of a novel or something or a fictional scenario one of the main environmental effects of generating electricity using fossil fuels is air pollution when fossil fuels such as coal oil and natural gas are burned to produce electricity they release a range of pollutants into the air including carbon dioxide sulfur dioxide nitrogen oxides and particulate matter these pollutants can have serious impacts on human health including respiratory and cardiovascular diseases and can also contribute to the formation of ground level ozone which can lead to smog and other air quality problems two marks I think it's it's fair to say on that two out of two for that question now we could spend a lot of time in the GCSE um questions just going through individual ones it gets a bit boring let's up the level we're going to go to a level now so AP level So High School uh level um the type of questions that um uh form the basis of exams for entry to University I think you all know what a level so notice if I just flick through this how often we're getting figures so for those of all those academics out there including myself we're going oh this is the end of assessment at the moment best thing to do is just make sure that it's all about data interpretation it should be about critical thinking data interpretation and I think at the moment if we want to bypass judge CBT then make sure we put lots of figures in there make sure it's all about data interpretation particle of mass m is oscillating with simple harmonic motion simple harmonic motion the bane of so many students lives but it's a very beautiful thing and it underpins so much of physics including fast way it's a Quantum field theory for one and quantum mechanics so this is a very traditional question as you can see it's one Mark let's see how it goes we're not going to give it a multiple choice motion period one Mark so it should be able to do this pretty quickly and actually for those of you who are physicists you're doing a11 you should be able to do that fairly quickly you don't have to think about courses you don't have to think about science you can't think about causes you can think about science you can't think about derivatives you can think about all of that but all you need is conservation of energy you know this all you need is conservation of energy to do this so thinking oh no it doesn't oh that's bad at the maximum no no no no no oh oh that's really bad the very first line is nonsense yeah it's interesting it gets the right answer but for the wrong reasons the maximum kinetic energy of the particle occurs at the maximum displacement from the equilibrium absolutely not so that's wrong and uh regardless of it getting the right answer if it were me that were marking this and the students put that down and hasn't crossed it out I'd be very very unwilling to give a mark in that sense because that's a complete misunderstanding of the Dynamics of simple harmonic motion so maximum Connecticut doesn't so basically imagine a pendulum going back and forth we started off from um a fixed position so it's not moving so it's got maximum potential let me stand up it's got maximum potential energy here zero kinetic energy then it's going to have zero potential energy here maximum kinetic energy this gets it completely wrong and what's interesting is that that first line is in the context of other stuff which seems broadly correct it's put the mats down relatively correctly but it doesn't get the physics concept which you wouldn't expect it to but that's worrying so for those of you thinking about doing using chat GPT to do your homework think carefully this is the maximum kinetic energy of the particle interestingly it got the right answer but for the wrong reason and for us so in this case I would be less willing to give the mark and I think a lot of physics teachers will be less willing to give the mark than in the first case where it got the reasoning correct and then it just screwed up um in terms of dividing that doesn't really matter well it doesn't matter if you send probes to to Mars or whatever you don't want to be about factor of 10. but the understanding was there and that's what you're probing in an exam the whole reason I started looking at this was because a student brought a coursework question to me typed it in and said so let me um what I want to do is ask it this is a tricky question this is a conceptually tricky question about quantum mechanics um and I know some of you out there won't have really done a huge amount of quantum mechanics some of you will I know there are a lot of um undergraduate students this is an important question consider the following statement okay I can't put I'm not going to put PSY and I'm going to call the quantum State why um consider the following statement the hamiltonian operator acting on any Quantum State y Returns the same state and its Associated energy eigenvalue e this is what the Schrodinger equation h y equals e y tells us explain why you agree or disagree with this statement now I tell the students in the in the quantum world class that um if if I if they come out of the module not understanding and not being able to get the right answer to this question I failed so let's see how it shot um GPT does explain why you agree or disagree with this statement and there are five marks for this in our marking standards so let's see what shot GPT does ah wrong okay we could go into details about just why it's it's wrong but the important thing is that that particular equation is called the time Independence routing equation it only works for certain States what we call situationary States the question very specifically says hamiltonian operator acting on any Quantum State this is exactly what I would hope that students do not tell me so zero marks chart GPT really bad feeling this definitely needs to go right back to the start of its quantum mechanics knowledge and get a drill I can point it to some very good online resources about quantum mechanics if it really wants them okay Quantum world coursework I'm gonna have to sell a new one next year um so it's actually about the um last computer file video we did about the superposition it's a lot of fun to do that one but it's about um particle in a box superposition of state and what the courseworks asks the students to do we we're trying to integrate more and more um Computing coursework uh into our physics degrees and because for me Computing is equally important if not more important than mathematics that that's a controversial statement but um write a Python program to plot the probability density for the N equal to one and N equal 100 eigenstates of the hamiltonian for the infinite potential world in the position representation and in the momentum representation if you're not at second year possibly end of first year undergraduate physics level that's not going to make a lot of sense to you but let's just see what chat GPT does does for this and I need to tell it what it is so I'll add a little bit this is not quite the question I'm just going to give it a little bit more clout of the hamiltonian further okay that's effectively trimmed down version of that question let's see how it does this requires a lot of conceptual understanding oh here's a okay okay good start okay it's using appropriate units okay that's fine good no it's done it that way okay thousand points I might be quite enough for something which is oscillating as high as that higher eigenstead but okay let me scroll down that's interesting is putting the energy eigenvalue in as well as the eigenfunction probability density then it's going to calculate the furry oh my God this is bloody hell oh good no it's it's yeah it's scarily good in in the in terms of doing the coding it hasn't quite got it's exactly it's made the exact mistake I'd hope the students wouldn't make um but in terms of the overall structure of the code that is pretty impressive and it's even the fft shifted it wow okay and then what's it doing is it still going it is so it's now explaining what it's done a program first defines the position of momentum grades and what's the probability density Ruby product okay so we'll leave it going churning away in the background that's pretty impressive um the one issue however is that the reason I said that question is it requires careful thought so what we have this thing as ever which even if you haven't done quantum mechanics you'll recognize this in terms of standing waves on a string that's what we're talking about important aspect of this is that in each case those functions go to zero outside the well what it's done is it's just assumed that those sine functions continue on outside the well that's exactly this mistake I don't want students to make because that doesn't give you the right momentum representation so it's performed as as well as a student that has a good understanding of coding and very good understanding of coding I would say but not such a great understanding really of the physics of this problem they understand a lot of the physics but not quite enough and they've made exactly the mistake I say they anthropomorphizing um they're um chat GPT has made the same mistake I would expect and would hope that many students wouldn't make so it's failed a sort of conceptual test here again if we could influence matter at the subatomic level by clicking our fingers we wouldn't have to spend billions on CERN we wouldn't have to say that and we wouldn't need this bloody thing no this is just nonsense this really is just nonsense and you know the argument will be made well science doesn't know everything so you know how do you know sure science doesn't know evident if science knew everything I'd be out of a job Brady would be out of a job because Everton would be out there but science knows some things

Channel: Sixty Symbols

Views: 588,444

Keywords: sixtysymbols

Id: GBtfwa-Fexc

Length: 16min 42sec (1002 seconds)

Published: Mon Jan 23 2023

Reddit Comments

[u/lacerik]

I definitely think this will have an impact on the types of homework that can be assigned in the future.

I agree with Dr. Moriarty that for the time being the best you can do is create tricky questions that require a decent understanding of the subject matter.

The age of homework that just tests rote memorization is probably coming to an end.