The limits of AI and ChatGPT: the common sense problem

so today I'm going to talk about chat GPT and AI so I've been a little hesitant to talk about this topic because I think by now Chachi BT is a little bit old news and everybody's gone through that phase of looking at chat gbt and looking at the amazing thing that it's produced so just this morning Jan lacun French computer scientist best known for his pioneering work in convolutional neural networks also Chief scientists at meta and professor at NYU colleague in a way I've never actually met him personally posted this very interesting video by Eugene Choi called why AI is incredibly smart and shockingly stupid so this is a great video I really recommend watching it but before you go away I just want to tell you basically what the gist of this video is basically what she's saying is that these large language models which is like watch Chachi BT is one example of lack what humans would call sort of common sense right so Choi gives this example of asking gpt4 this question I have a 12 liter jug and a six liter jug um I want to measure six liters so how do I do it okay so if you want to measure six liters and this is a six liter jug then obviously you just fill up this bottle and that's the end of it right you don't you don't even need this one right but what chat GPT says is to say okay fill up the six liter jug okay fill this off okay pour this one into here into the 12 liter jug um then fill this one up again now pull this one into the 12 liter jug now the 12 liter jug is full imagine this one is full now there will be six liters of water in the six liter jug no the picture should be should be empty right I mean you just emptied it here uh it doesn't really make sense this is one of these examples that Chachi BT is giving the wrong answer however it suggests something a little bit strange here because it doesn't seem to really understand anything about how water and jugs kind of behave in general right it's basically Troy's main thesis is that yes there will be improvements to these models in the future but what if there's something really kind of missing or something that we might be able to actually even do better rather than by just simply training bigger and bigger networks which is basically what you know people are doing now so what I found interesting was a kind of a comment to this video uh to Yan's original purse which went something like this so Henry couch says unfortunately for yejin and most AI researchers including myself so I think Henry is a AI researcher uh turns out that common sense is the easy part uh it is 90 done with gpt4 and will be 100 complete with gpt5 however deep scientific knowledge making fundamental discoveries in biology chemistry and physics is harder within the ear llms large language models will work as valuable assistance in every research group coming literature linking data sets suggesting hypotheses and designing experiments the research at least the most significant research will however continuing be led by humans whose level of insight cannot be replicated but give it a decade and that too will change but what what Henry is saying here is basically more or less what I was kind of thinking myself after playing with chat GPT and you know first being astounded like everybody else going okay but actually maybe this thing does have its limitations and it touches upon a question that I think is has been on the minds of everybody really who's played with this this year in is which is that you know will AI now kind of replace all these jobs that are out there right and chat GPT and similar AI models really seem very very good at making things like essays and reports and poems and in fact kind of any kind of document right shortly after chat GPT came out at NYU incidentally there was you know quite a widespread discussion mainly by the humanities professors about how they're going to deal with the emergence of chat gbt because you know you're going to have all these students that are just like uh you know type please write an essay about you know this and that topic right which is just their assignment and their assignment is done in like two seconds this is you know a major issue and I mean each time you do it you don't even get the same result and you know you have all issues with sort of plagiarism and that right so um they're certainly right to be worried I was personally less concerned because you know it hasn't got to the point where solving physics problems I mean it doesn't even have the right interface but you know perhaps this will happen eventually too um but my general answer to this question of whether AI will be replacing much of human uh work uh goes really something like this so I think it really depends upon how generic your work actually kind of is right and so just to understand my point let's just look fundamentally at what a neural network actually is right as we know so neural network consists of a bunch of nodes which each node represents a binary variable interconnected with weights for a particular input there's a simple mathematical relation to determine the output for a given set of inputs and weights so it goes like in this diagram you have all the input variables you have some particular weights which Define the actual neural net and then you add them all together just basically using these weights and the values and then you put it through this activation function which is basically kind of normalize it to the right level and then you have your output and so you know when you have this large the circle deep neural networks then you just have many of these with multiple layers and so what you have at the input there finally goes to Output okay so probably most of you know this but you know this is just background just in case you know you're not familiar for a given task like pattern recognition we always need to train the neural network right and so this basically means you getting these weights these W's are in this formula so this is basically the equivalent of programming the network and the great thing here is that unlike traditional programming where you go to sit down and like code something up instead of writing all the different conditions in terms of recognizing what actually is a cat essentially that programming work if you might call it is all sort of done for you in the training process right so this is far better than somebody trying to work out some elaborate rule uh for things like handwriting which is the classic pattern recognition kind of task and so you know it does very well at these simple examples that people have known for decades but now you know we're getting to the stage we're doing much more complicated things okay so why did I tell you about all this super basic stuff so well the point of it is that at its Essence these neural networks are actually a lot like fitting functions right so we're all familiar with fitting functions you have a bunch of data points and you want to fit it with some smooth curve usually this smooth curve has some kind of functional form that you derive it's got some parameters and the idea is to optimize these parameters so that the curve goes through these lines smoothly so basically what we would call the training part of the neural network is simply this kind of minimization process where we try to match the fitting function to the data points the best we can so this is obviously a great simplification but you know conceptually at the core of it this is basically what we have okay so using this intuition what does it give us basically you know these fitting functions usually interpolation works very very well so basically this means that if you've got data points that are on this line and you want to know something between the data points between these points you get a very very accurate answer but extrapolation on the other hand does not work that well you don't know whether this curve is going to go on the right hand side here will it go down or maybe it'll go down you know and sort of turn around and come up you know it's basically impossible to tell without having any data so basically the point here is very simple so the less data there is the less reliable the network is I mean it's really as simple as that so going back to GPT there's a very good reason why it's very good at making these kind of generic maybe high school or university level kind of essays and documents as we've seen so there are a lot of examples of these types of things that you can train these AI networks on and so if there's a lot of things to train it on it's not so surprising that it is able to kind of reproduce them because it's basically seen a lot of examples of it right so the really the answer to the question of you know whether AI is going to really replace your job here kind of depends upon how generic your work actually is right so if your job is to write something where there are numerous huge numbers of examples then yeah maybe maybe an AI will be able to do that as good as basically what you're doing so going back to Henry's Point anything that is at the Forefront of human knowledge is basically by definition something where there's not not going to be very much data to train any kind of network on right because that's basically what the Forefront of human knowledge means it means nobody has done it and if nobody has done it well where are you going to get the data to train things on so if you can't train it then of course AI won't be able to reproduce it so I don't know about in a decade where the Forefront researchers will be replaced which is his last comment there but the rest of it I basically more or less agreed so this is why I was a little surprised when Yan actually replied and said that I totally disagree with you and agree with Eugene I made that point in all my talks in the last seven or eight years and it's still valid GPT whatever has very little Common Sense the vast majority of human knowledge is completely non-verbal and absolutely not captured by llms until we have models that can learn internal models of the world from high bandwidth non-textual channels such as video we will won't have machines with as much common sense as a house cat let alone a human child but I think actually the point of the disagreement is not to do with the point that scientific discovery or anything that is truly creative or importantly basically anything that's novel is going to be difficult for AI models I suspect that Yan would sort of agree with that point at least in this point in time the point Jan I think is making is the first part that even Common Sense will not be complete by say gpt5 right the point is that current large language models are taking Text data and this is actually quite a biased set of things to train AI on this was only done for things like chat GPT because you know that's very readily available right things on the internet which are basically what these things were trained on we only tend to write things down as humans that are kind of significant and or unusual or kind of difficult right things which are sort of the bleeding obvious you're not going to really bother to write down and so you can see why this previous example that we had of the jugs kind of failed because tried to generalize this kind of puzzle that you often see in mathematics or computer science how do you use these two drugs to measure something which is neither the the volume of these two it's obviously looked at many examples of this and then tried to apply that logic to this problem because it sort of recognized this question as one of these types of puzzles when in fact it's not even anything like that puzzle right so additionally Troy says that some abstraction steps may be required so that it's not just training increasingly larger and larger networks which is basically what you know people are trying to do now so let me finally just go back to my friend Paul um where the post that he made kind of recently he says an atheist friend who thinks the mind is nothing more than the brain I believe this is this is me uh because we have these types of discussions a lot uh says that AI will never be as imaginative or creative as the most intelligent creative humans so this is basically the point that I guess I was trying to make I think never is a strong word but that's more or less the point that I was making it will always have a degree of mediocrity uh even as it improves in skill and ability compared to the brightest humans of the day I was thinking about this and it occurred to me does this kind of thinking implicitly concede that there is more to the human mind and intellect than merely the brain that the reductionist physicalism view of the mind brain problem is in fact inadequate that there has to be something more to the mind and Consciousness than simply neurons firing if not then surely AI can achieve a similar feat perhaps even Superior of imagination and Innovation as the brightest human Minds since it is effectively merely mimicking the way the human brain operates so a bit of background my friend Paul here is a devout Christian and you know we have endless conversations about you know the existence of God and things like this um so I kind of know what he's thinking when he's writing this basically this question that he's really trying to lead me into is to say that there's something kind of special that cannot be explained by science that there might be something possibly even Supernatural about humans and maybe the world and the universe that science will never be able to touch the way that I think I would respond to this is that again this is sort of pushing the boundary between what is the explained and The Unexplained so these what these AI models have really shown is that actually AI models are very very good actually at mastering kind of the human language like it really can sort of speak English and well actually any other language really quite remarkably well so regarding the point that I'm a reductionist physicalist meaning that basically I don't believe that there's anything more than the molecules in our body and everything should be explainable by basically what we can observe and what we see in science um you know I think I you know it's accurate to say that I probably am but I think there's a sort of a missing Point here that you know the models that we have are already somehow complete but I don't think anybody would assume that uh you know this is the final model of AI and basically everything else is simply an extension of this we've been building these models for decades and decades and who knows you know what kind of improvements there are so as Troy says in her video perhaps there are other mechanisms that need to be incorporated to have true artificial general intelligence so there's always going to be kind of the front line of the research and knowledge and this is how science always advances solving these problems that come one at a time so to think that this is a static line that's not going to change is I think a kind of a strange assumption that is being made here and it's similar to how sort of creationists argue that science will never ever be explained how life actually came into being some people would say that oh Evolution explains everything you know I'm certainly not in that camp where you I think you we can perfectly understand how life for example originated I think there's a lot of mystery to that and probably a lot of research that still needs to be done unless basically you can actually make life in the lab yourself I don't believe you can really understand how it really works if you go on the track record of basically what have we've achieved already is sort of humanity of the kind problems that we have been able to solve then I don't know my bet is that eventually things will be all explained but we're just not at that level where we can understand everything perfectly yeah okay so that's all I wanted to say today so thanks for watching and we'll see you next time

Channel: NYU Quantum Technology Lab

Views: 2,842

Keywords: artificial intelligence, machine learning, chatgpt, openai, AI, mind body problem, mind body, GPT-4, GPT4, yann lecun, large language models, LLM

Id: w9MSsnoTSJs

Length: 17min 20sec (1040 seconds)

Published: Mon May 01 2023