Dissecting The Extinction Threat of AI | Yoshua Bengio | Eye on AI #128

very hard in terms of your feeling of you know your ego and and how you can feel good about what you do to accept the idea that what you've been working on for decades might be actually very dangerous for Humanity in a 1955 paper titled the computer and the Brain John Von Neumann wrote that the possibility of replacing the human brain by a machine which will be superior to it in any or all respects is not excluded by any natural law that we know it's therefore possible that the human race may be extinguished by machines Von Neumann's warning has since been echoed by many other AI experts from Stephen Hawking to Elon Musk the concern captured the public imagination earlier this year when the future of Life Institute issued a letter calling for a six-month moratorium on large language model development saying that the development of artificial general intelligence could lead to human extinction then in May the center for AI safety issued a statement warning that the development of artificial general intelligence could pose an existential risk to humanity many who had not signed the future of Life letter signed this one including Jeff Hinton and open AI CEO Sam Altman Joshua bengio who signed the future of Life letter had spoken on this podcast before about the threat but I spoke to him again specifically about the extinction threat whether it's a useful characterization or simply adding anxiety to a generation already troubled by looming existential threats such as climate change so huh just a moment sir yes I apologize for interrupting you but I wanted to remind you that Oracle is offering a full implementation of netsuite thanks for reminding me please continue okay you can go back to your station thanks um yeah uh uh Beauregard reminds me that the first time in netsuite's 22 years is the number one Cloud Financial system you can defer payments on a full implementation for six months that's no payment no interest for six months to take advantage of this unprecedented offer go to netsuite.com backslash ionai that's www.netsuite.com backslash I on a i e y e o n a i all run together now here's yahshua hey yahshua I wanted to talk to you about the uh your your essay on how an AI could become Rogue and since then this uh Extinction debate has ramped up on my Twitter feed there was a long thread by somebody with a blue check mark that doesn't mean much anymore but uh basically bemoaning um you know the generation we've already destroyed uh because Extinction is inevitable uh now that we've developed these uh these language models uh and and that just I mean it just to me that that debate uh which is bleeding into the general uh public discourse has gotten out of hand but uh you know your your essay was really the first concrete response I've seen on how things could go wrong uh so I I wanted first for you to just present that scenario and then I'll ask some questions um I don't know if I told you last time but I'm also writing another blog post which um lists pretty much all the um pros and cons of that discussion that I've heard so yeah let me summarize the debate in the way that Max tegmark did it near the end um so he said yahshua that we we are on a river and you know going down the river and on our Collective boat and Joshua hears uh some waterfalls Downstream um and Melanie says there is no waterfall or you know if there is one it's going to be maybe in 300 years so we shouldn't worry and Jan says um yeah of course there's a waterfall we all agree on it's about you know a few years to a few decades away but we'll figure it out when we get there so that's like a high level kind of description of each one's positions um max also talked about some existential risks that I haven't been like really covering myself or talking about in what I wrote that has to do with a slow um disempowerment as we get more and more dependent on the eye and the eye builds up its power at some point um it becomes very very hard to pull the plug even if we could um because we're completely dependent on it and it you know the society would break down and and yeah so so that's another kind of scenario which is a sort of a interesting to think of and some people in the AI safety have been also thinking about but I've been mostly talking about the simpler scenarios that I talked about in the Rogue AI essay so they're very very simplest one uh doesn't even require like some very fancy knowledge of machine learning or anything or reinforcement learning or AI safety it's just that it just assumes that in the near future which as I said there's a lot of uncertainty like is it years or decades um we we know how to build a superhuman AI and if that recipe is publicly available and even worse if the weights of a trained AI are available then it becomes very easy for a huge number of people um to issue instructions that if they were executed by the AI could lead you know terrible catastrophes and in the worst case could lead Extinction and and I you know I can explain maybe with some scenarios what that means yeah so so that that's um I feel like I I haven't heard any like serious criticism of that I mean maybe the the strongest um counter argument is Yan saying oh but don't worry we'll build we'll use uh the progress in AI in order to build um counter you know measures AI that they're going to help us uh fight off the road guys and because there will be more good AIS and bad AIS you know the good guys will win um maybe and I think we should definitely do that in fact I'd like to work on something like this but I don't think it's a silver bullet we know in kind of military situations or conflicts that sometimes the attacker has an advantage because the attacker can do things silently um and sometimes there is not a lot of time to react or to prevent uh you know some large amount of damage then I talked about the what I call the what I now call the Frankenstein scenario so so there is this sort of naive scenario where somebody naive but but also but plausible where somebody just out of anger or uh you know conspiracy theory you know crazy beliefs or for military reasons issues these highly destructive instructions um a slightly um kind of less obvious but but also fairly likely scenario and that enlarges the set of people who would do this is someone issues instructions or you know designs the AI so that it would have its own self-preservation Instinct in other words one of its main goal is to survive just like us and every other living being which essentially means we end up with a new species on earth a new species that would be smarter than us and you know the past of more powerful or smarter species um acting in ways that have driven to Extinction other weaker less smart species isn't very we're sharing just in the last few hundred years humans have driven almost a thousand species to Extinction that's the ones that are you know that we know about and that's not to mention our cousins hominids that are of all disappeared I don't know why but you know uh be surprising if if Homo sapiens didn't have a role in this and and when we drive a species to Extinction it's not necessarily that we want to destroy them it's just that there are like their destruction or their disempowerment uh is a side effect of us trying to achieve our goal um you know we want more land more profit more food and that's mostly the way that we've driven a lot of species to Extinction or some species still exist but they're completed as empowered so gorillas are still there we're probably going to keep them around but how much agency do they have you know um over their environment it's all controlled by us um I don't think humans want something like this for for Homo sapiens yeah can I just interrupt for a minute because uh the yeah certainly these scenarios have been uh in the public space in in science fiction uh novels and and movies of for decades what happened that that uh and and certainly you've acknowledged uh sort of distant risks in the past people have talked about it but the consensus by you and Jeff Hinton and others was that that is so remote we don't have to worry about it right now there was this dramatic uh uh step function you know where things uh progressed very quickly with the Transformer algorithm uh but why now I mean that that generative AI certainly their abuses uh in the current uh ex current level of technology that are possible through different disinformation and that sort of thing or if they're hooked up uh to the internet you know you could construct scenarios where they could do do damage to uh to things that are are available through the internet but uh but why did you suddenly decide that this threat is near term enough that we need to to act on it um in a few words we we've reached this stage of essentially passing the Turing tests meaning that uh it's hard to distinguish if you're talking to a human or a machine if if you're exchanging with gpd4 I mean and right now if you ask it it will tell you that it is a machine but but it could easily be the case that it doesn't do that so that was of course thought by Alan Turing himself um uh many many decades ago at the beginning of computer science that is a way to evaluate whether we've reached human level intelligence now I don't think we have because if you take much more time to uh kind of try to trap gpt4 you can find some issues and but they're like I don't know hundreds of billions of dollars invested to fix those issues because gpd4 and you know chatbt have been working so well and in fact the work that I've been doing for many years is all about fixing those issues it's adding the system to capabilities to deep learning right now deep learning is very good at system one capabilities which roughly correspond to intuitive immediate reaction you know behavior that you do without taking the time to think about it and that require a lot of practice in order to hone in system two allows us to do things where we don't even need a lot of data we can just think through reason our way to find solutions to problems and we have a sense of where there are some statement is is true or not um and we can reason with causality and things like this but this means progress on system two uh and other issues that people are working on [Music] it's hard to say if we you know maybe we're close to a breakthrough on on these aspects and maybe just two three years or maybe it's 20 years at what there may be other obstacles we don't see right now but but from my vantage point I'm very concerned that we're not very far and I also know that Society takes a lot of time to adapt to come up with counter measures uh regulation International treaties everything that I need I think will be necessary in order to reduce those risks and finally you might have you know you asked me like why now why didn't I like think about it before why didn't Jeff think about it before well so we thought it was so far away we didn't need to worry um but also there I believe there's a psychological effect which might be still at play for a lot of people it's very hard um in terms of your uh feeling of uh you know your ego and and how you can feel good about what you do to accept the idea that what you've been working on for decades might be actually very dangerous for Humanity because there are people who've been working on AI safety for you know at least a decade and others longer but mostly it's been in the last decade a lot of work has been done I mean it's very very marginal in the machine in the community but still there are people who have been worried for for some times and have been writing papers to try to think about different scenarios and countermeasures and so on um people like Stuart Russell for example who wrote very nice book human compatible about that and how we might potentially build safe AI systems so um I think that I didn't want to like think too much about it and it's probably the case for others uh you were saying uh that you could walk us through the extinction event and it's really the word extinction that that bothers me as I said on Twitter they're they're apparently credible people who are already sort of bemoaning uh the extinction of the human race because it's too late uh to stop this development uh and and when that gets into the public uh it it it just spreads fear and yeah you know there are a lot of wonderful things for sure yeah and there are a lot of wonderful things that AI can do I mean in terms of Extinction events you and I have talked about this we are facing an Extinction event with climate change very little doubt uh those the the the systems that contribute to climate change are so complex we need the the pattern recognition capabilities of AI we need the reasoning capabilities of generative AI to to help solve those problems so from my point of view it's AI is probably our best hope uh to to avoid Extinction but when you tie AI to Extinction uh you know yes it it seems to have woken The Regulators up but it it creates a very uh a strong headwind for uh for for a the positive uses of AI yeah yeah um I'm totally with you on this I would just like us to be wise and walk this fine line where we can get the benefits or at least a large number of them but avoid the catastrophic outcomes so yeah I'm not saying it's going to be easy and just like uh being submerged by fear isn't going to help either um here's the thing about desperation I don't think that we're cooked um let me take the the climate change analogy climate activists could feel desperate and just drop it drop the ball but they don't we should have acted 20 or 30 years ago we we did it but now we are today what can we do to move things in the right direction and there's until we we we have agency which we do right now it is possible to reduce those risks um that's what I'm trying to do um and yeah you have to be careful not to create a panic but but trying to steer something positive uh that that is going to help on both sides there's there's this concern about super intelligence uh right now uh what we're dealing with uh is a system that by ingesting massive amounts of data has learned to product predict the next token in a series essentially when applied to language that it predicts the next word there's there's no uh deeper intelligence uh there there may be uh the illusion of uh of even sentience but you're involved in building these systems uh and you know that there there is no uh yeah there's no ghost in the machine thank you uh and the idea that that we will arrive at the point where we can build that and no one knows how to do that uh there's certainly uh people trying it just seems the scaling up of Transformer models uh was uh surprised everybody at at what the system could do but that's still a huge leap from building a machine that has uh an independent intelligence uh and agency so no actually not so you know um GPT shows you can take chat GPT and create a thin layer around it a wrapper that provides it with Agency for example to act on the internet and you say that you know there's a huge gap well how do you know I I don't see that I see that it could be a short while I don't know for sure and and because I'm not sure I don't want to like take the risk that oh it's you know ignore the potential problem yeah um also there's no ghost in the machine in your brain either as far as I know yeah but there are many many many more structures in my brain and in your brain than a neural net yes that's true it's true but uh it looks like so the kind of intelligence we're graduate building is not a copy of our brain right you know Evolution has come up with our design through many torches uh turns it's not clear that you know you need uh that that it's the only solution to intelligence um personally I think that we've solved a large part of the problem and you know system two is something that came fairly late the scale of evolution um we're not sure of course but but you know it looks like uh something that might not require that many bits of information that's been added on top of the all the things in evolution I found before so that means we may not be that far and we have already some plans to to deal with that what in your research I saw recently you said that you're going to shift your research to safety uh what in your research uh are you are you abandoning that uh uh that would lead toward more powerful AI what I would like to avoid right now is to put out information that's publicly available um that would help to bridge that that Gap that remains too quickly I will continue my projects in AI for social goods which is about very specialized systems that you know don't understand how society and humans work but maybe some very small part of it um and and like I need to take the time to think about more precisely what is my best course of action the uh you you you talked in the the paper uh on on how a rogue AI could arise about open source and the danger of Open Source uh and and you talked about uh you know genocidal humans getting a hold of an AI and and doing right something destructive with it uh on in in both of those cases even with open source code you need at least today a tremendous uh Financial Resources uh to Marshall the hardware well you don't need to Marshal the hardware yourself you're right it's available through the cloud but you still need to pay for it uh no no if if you if the if the weights are shared then it's very cheap to um you know fine tune it put the layers you want around it so that it's going to be specialized to the task you want which may be some some malicious objective for some nefarious objectives so you're saying if there's an open source pre-trained model yes that you can copy the weights uh that's already the case so I mean but not not with you know uh superhuman AI but it's already what what essentially Facebook has done with with uh open sourcing uh models yes including the weights yeah and then on the agency side uh you have these rappers like Auto GPT that that do give uh these large language models access to the internet however and goals right it turned them into goal directed uh AI systems yeah although I I've played around with auto GPT we talked about that last time and it's it's not as straightforward but that's not the point the point is that in I mean people who know about reinforcement learning know that it's easy to design something like this now the the reason it's not working that great is because Auto GPT itself I mean uh GPT itself is not that smart but what about 235 10 years from now yeah uh the uh you also uh in your paper and I appreciated this talking about uh corporations as a form of artificial intelligence that that uh you know do a lot of bad things in the interest of a goal that's not aligned with uh Society um uh and so a lot of this when you say work on safety are you talking about uh working on on the so-called alignment problem how you instill uh human aligned values and goals and prevent um that's one way so the other way that I've been talking about is to try to design um AI systems that don't have agency by Construction um now uh and and our trained not to please us like my current child GPT um but to be truthful in a probabilistic sense uh to understand how the world works and then to answer our questions based on this knowledge um now these could still be turned into agents the same way that you know Otto GPT works so you you need some other differences and and I think with the combination of both like technical aspects and uh societal aspects like you know who has access what kind of governance you put around these things um who decides what we do with this and and so on um I think we can considerably reduce the risks um the EU has passed its uh though not past but uh the parliament has submitted its first draft of its AI acts Canada I'm not sure where they are in the process but are close to doing the same uh do you think these regulatory efforts answer your concerns um they're stepping the right direction but they haven't been designed for um catastrophic risks so can you walk us through a us an extension Extinction scenario I mean that's what what everyone's a little uh alarmed about a little uh when when people like you talk about Extinction uh so if you could just give kind of one scenario of how that could happen and uh understanding that as you call it there's still a gap between uh the AI that we have today in in the AI that would pose that sort of a challenge one scenario um that that I think about is the this Frankenstein scenario where you know we we have a tendency to to want to build AI at our image and in fact a lot of people today this is the right thing to do problem is um if we do that means we are giving these machines the same kind of self-preservation instinct um that other you know that every animal has and with that come as a uh important sub goal if you want to survive first of all you don't want to be turned off that's the first thing so um if we ever have the intention of to any one of these Systems off and it it understands that it's going to do its best to prevent us so you can see already a conflict here um also the best way to survive is to figure out how to control your environment I mean all animals do that to the extent it can but now we're talking about an entity that is smarter than us so we are part of its environment that means it wants to control us yeah but but but we're not talking about embodied intelligence there's no physical manifestation that that that comes easily so first the kind of systems we're building now and we know how to build uh probably for the next few years you're right don't have a body they have actions though in the real world which are the for example the dialogues or the images or the videos that they can produce and potentially if somebody connects them to the internet all the things you could do on the internet um so how could they um you know do something in the in the in the non-virtual world well they can convince they can manipulate people they can pay them for doing some things you know this is you go on some websites and you ask people to do things legally um or you can pay them to do illegal things like organized crime is happy to do things without asking where the money comes from and these different uh people can be contributing to a you know a bigger project and looking at their piece doesn't seem so terrible right um at some point you could imagine that these systems will be able to design robots that are you know better than the ones we and then have their own body as well I mean it's not the same thing as our body because we you know if our body goes away we are we're finished but but these systems can reproduce themselves they you know if if one body dies it's all right they're you know copies of the code in many places um in a way they are Immortal unlike us you know our body is going to eventually uh not be functional anymore they can copy themselves is is many times as necessary um so the the the the yeah that's the kind of scenario by which they can take control now how could that lead to you know what kind of action is good they do people have been talking about two main vectors and maybe they'll think of something else one is bio weapons and the other is cyber security attacks and by the way with cyber security you could gain control of uh things that that have an importance in the real world like uh your energy infrastructure or weapons bringing down are Communications infrastructure you know combined with other things that could be terrible so for example if you combine that sort of thing with uh the release of a bio weapon in other words new pathogen that would be more virulent and viral than anything that we already know it's quite conceivable there's already research going on to use AI to better understand how our cells work and I think this could be extremely positive Revolution but it also means that we might you know reach a point where AI systems could design new living beings starting with viruses that that could either help us a lot or harm us a lot yeah so it's it's really an issue of uh agency of whether or not we allow these systems to have agency as you pointed out there are already these rappers of on uh on GitHub that uh that are giving uh GPT for agency uh although as I said it really doesn't work because gbd4 hallucinates and auto GPT just compounds the hallucination and next thing you know you're stuck in some endless loop or the thing crashes um but as you said that you know it's early days those those problems will be addressed uh so would this mean uh on the regulatory side outlying uh allowing these uh systems to to have a live connection to the internet I mean uh maintaining them as uh air gapped systems is that the solution or or do you think it's more about uh just regulating um the the the the the systems themselves I mean how large they can be or how capable they can be so first of all you know I've just started thinking about these questions I I don't feel like I have all the answers at all and I think we need as a priority to spend more of our brain Cycles to study these questions um both on the side of AI safety like the computer science side and on the policy side but it seems reasonable to uh consider some forms of Open Source and sharing of models to be something we want to limit when when these systems are dangerous or potentially dangerous of course um but most of what we do in AI right now is is it's not dangerous at all right so we're talking about systems that don't exist yet but we don't want them to be too easily accessible although the flip side of that has I mean we've you've been through all these arguments endless times at this point but the flip side of that is is you have a a small group of people holding uh tremendous power and that was the argument behind setting up open AI in the first place ironically uh was because Google uh had had you know this proprietary Tech that was extremely powerful and there was a thought that it should be open sourced uh but of course open AI then came up with extremely powerful Tech and and closed it as well so um yeah what do you say to that argument that that you're then concentrating me who gets to decide who controls but it's just the Whoever has the most money to hire the most Engineers well not necessarily um we could set up governance systems by which these organizations work for the public um so concentration of power is something that's already happening and that we want to avoid absolutely I agree so we have to find ways to be safe and Preserve democracy because democracy is all about you know not concentration of power how can we do that um so one one thing I've thought about and several people have talked about you know a CERN of AI so the idea is actually I think there should be several but to have organizations that are non-profit um and under very strict um governance to make sure that we avoid the arms race between companies or between countries that is going to happen if we do nothing um so it's not just concentration of power it's also the arms race in other words companies or countries competing with each other and cutting Corners which could be dangerous from a safety point of view now you know there nothing is easy and and you know I'm not sure uh if governments will be eventually convinced that this is uh the right path but that's my opinion so what I mean by this kind of governance is that to avoid the arms race um it would be something in which the you know as large a coalition of countries as possible and especially those that are opposed to each other like the US China and Russia are part of that governance and and the PO they so that what is going to be done with that AI is going to be um managed so that it's not targeted at um you know hurting some other country or something like that um yeah and and I think that if we allow these kinds of super powerful AI to go on in companies they would also have to uh obey very severe Protocols of safety yeah with with people like you and Jeff uh peeling off from uh the the basic research uh do you think on the one hand that that progress toward AGI or toward the this super intelligence that that we're concerned about will slow or do you think that there there will be a parallel tract where uh we're working on safety and guard rails and policy and Regulation and on the other track there's you know yen lacun is not slowing down on his world model efforts right I think it's going to be parallel tracks for a while um I um I think that even if Jeff and I didn't contribute to the public advances um in AI there are so many people working on this so much money invested that you know we're going to continue to move forward but and that's exactly the reason why I think we need these um tightly governed organizations that will build hopefully the first superhuman AI but in a safe way that can help us mitigate potential attacks from Rogue AIS coming from the uncontrolled efforts yeah but really these are all speculations right I think it's very early right now I'm just like drawing these as potential scenarios I think more people need to think about it and exchange and um these scenarios need to be evaluated and discussed uh across many stakeholders yeah I I saw uh Andrew had a conversation with Jeff Hinton uh and and one of the things they said is that right now the AI Community uh is divided on this question uh and in certain quarters as I referenced on on Twitter uh that debate is becoming a pretty heated pretty angry uh and and what needs to happen is the two sides within the research Community needs to to come up with a consensus view because otherwise it leaves uh the public and and presumably Regulators confused uh you know who do they listen to yeah it would be nice um I don't know if it's going to happen uh there's always been issues that divide scientists and which policy have to take decisions thinking about climate tobacco for a while at least um the way that I think about this if I was you know um in government is that there are scientists who think this is a very dangerous issue there's scientists who think we should not worry about it we don't know who's right but the stakes are high if we ignore the problem and it turns out you know the the concerned one we're right so so I think we need to apply the you know cautionary principle here as a society right now that means regulation are you working with the Canadian Regulators who are with any of uh with the EU on uh or the US for that matter on on advising on on regulations or legislation well I mostly been working with a Canadian government but I've been talking to um senators in the U.S as well yeah and uh so you're just to sum up uh you know we don't know how distant this threat is but uh the threshold of of capability has been reached where it's time you think to start working seriously on these problems uh but without scaring the public about imminent Extinction which I think is what's creeping into the public discourse yeah it's not clear how to do that yeah I mean the strategy um the way I try to present things is not to focus on extinction risk but just catastrophic risk which is already bad enough think about nuclear weapons I mean the most part they're used Will Survive but very badly climate is likely like this as well probably adapt that Society could be badly destroyed many people could suffer and die that's it for this episode I want to thank yahshua for his time if you want to read a transcript of this conversation you can find one as always on our website that's www.eye hyphen on dot a i and and remember I don't think the singularity is near but AI is changing your world so you better pay attention foreign

Channel: Eye on AI

Views: 24,253

Keywords: AI, Yoshua Bengio, superhuman AI, AI ethics, AI regulations, Future of Life Institute, Centre for AI Safety, deep learning, system-2 capabilities, AI advancement, AI risks, societal dependency on AI, AI and climate change, AI autonomy, AI misuse, AI public awareness, existential threat, humanity’s future.

Id: 0RknkWgd6Ck

Length: 48min 49sec (2929 seconds)

Published: Thu Jul 06 2023