The Emerging Global Agendas for AI Safety: The Policy Challenge of AI Safety | Hoover Institution

[Music] I want to welcome you to the beginning of two days of discussion about Ai and Society being hosted here at Stanford University on the first day on the first afternoon the Hoover institution is organizing this conference to focus on the issues of AI safety tomorrow uh Stanford Hai Stanford center for human centered artificial intelligence is going to hold an all day Symposium that's going to broaden the aperture out to all the issues of AI and Society I want to take a moment for those of you who uh have not yet fully absorbed the landscape to understand the elaborate landscape of AI work here at Stanford of which we're a part there is uh there is Stanford Hai which has this wonderful Symposium tomorrow Stanford Hai again that's human centered artificial intelligence includes things like the center for research on Foundation models and a number of other Affiliated centers also at Stanford there's the Stanford center for AI safety they tend to define AI safety a little differently than we will today they're concerned with things like Research into the safety of autonomous vehicles for example then there is the Stanford cyber policy center with which maricha is closely involved D there's also a Stanford AI lab nicknamed s uh which is so there's a whole U set of Institutions here the Hoover institution is an institution that devotes itself to public policy and public affairs domestic International economic education and including science and technology in fact uh Hoover has recently been kind of a co-founder founder of the Stanford emerging technology review which Amy zert leads at Hoover along with others that's designed to help the policy Community stay AB breast of the most important developments in emerging technology so here we are today we're at Hoover we're working on the policy issues and we're working on the policy challenge of AI safety our first speaker to kind of kick off the conference is maricha Shaka uh maricha is uh is here at Stanford as the international policy director at the Cyber policy Center and she's also a policy fellow at Stanford hii she's been working on the intersection of technology and policy for a number of years it was a key part of her portfolio when she was a member from Holland for the European Parliament she was in the European Parliament for 10 years representing the Dutch liberal Democrat party she's involved with the UN scientific panel on emerging technology and Ai and a lot of other efforts and later this year she's going to have her book coming out from Princeton University press called the tech coup how to save democracy from Silicon Valley so with that Mara over to you thank you so much for bringing us all together um and for organizing this very timely discussion on governing for AI safety after having been here for four and a half years at Stanford it's my first time speaking at Hoover so that is a special moment to remark as well it shows how many different um parts of the campus don't always interact and I appreciate the fact that they do today so you asked me to speak a little bit about global governance of AI uh in 15 minutes but to make that manageable I thought I would focus a little bit on the European Union then a little bit on the United Nations and end with a question that I think is under addressed when we talk about AI safety and some of you may see that as a bit of a provocative question but we'll get there the EU as I'm sure has not escaped you has recently adopted the first binding comprehensive law on governing artificial intelligence it's the first such law in the Democratic World while of course um the US relies mostly on existing law for understandable political reasons and while China has binding regulation It Is Anchored in a completely different political system set of values that is not the same or even comparable to that in the Democratic world now the EU AI Act is designed to focus on the impact of the application of artificial intelligence if there's one thing you remember today I think it's important to keep that in mind so it does not look at the technology itself which would be another way um to look at governing AI although there is one exception that I'll talk about later the law then proceeds to define a spectrum of risk emanating from the use of various AI applications and recommends and requires mitigating steps to be taken by the developers of these AI applications accordingly so it's proportionate to the level of risk that these mitigating measures kick in what is important to keep in mind is that this risk based model also incorporates whether an AI application violates people's rights I've heard beautiful presentations ju opposing the risk based approach to AI governance to the rights-based approach to AI governance but when it comes to the European Union these two are very much intertwined and I'll illustrate that with some examples and even if the focus of the eui ACT is on AI risk the majority of AI applications are anticipated not to present any risk at all or when an application is in scope it will probably fall under the minimal risk category which is anticipated to be the greatest now let's take an example of what this minimal risk category looks like the many interactive or generative AI applications where manipulation could play a role think about chatbots for example and such minimal risk AI applications are then subject to transparency requirements meaning that the user must be informed about whether it's interacting with an AI chatbot or that synthetically generated content should be marked as such and watermarked when that's possible high-risk applications then fall into several categories either an AI system that has become a component of a product that is already subject to safety standards in need of third-party assessments so think about critical infrastructure or healthc care where there's a rich body of Regulation and then if you enter AI into the use of these systems those regulatory oversight processes should incorporate the safety risk emanating from AI but there are also additional obligations for the providers of these systems to include Conformity assessments and detailed documentation of the capabilities of systems so it's all about understanding better what these systems are capable of and then there's the most far side of the spectrum unacceptable risk which is a very narrow category of AI applications for systems that are used to classify individuals based on Behavior so think about social scoring or when we think about predicting um characteristics or behavior like crime prediction or subliminal manipulation manipulation so subtle that the person um exposed to the application can't really perceive it and could thereby easily be manipulated for example in the context of Elections but think also of realtime biometric identification in public spaces when law enforcement uses them although there are some narrow exceptions or systems that actually try to read the emotions of the user in the workplace or in an education facility so this gives you a sense of the spectrum of risk that the eui ACT has as its foundation but I mentioned there was one exception which has been hotly debated so you've probably read about it in the newspaper but Midway in the legislative process when generative AI broke through a whole discussion emerged about what the EU should do with Foundation models or general purpose AI systems as they call it we call it Foundation models here at Stanford and so many of the political leaders believe that these systems could not be excluded from this first comprehensive AI law and therefore they were included meaning that models of a certain scale require extra safety measures including red teeming incident reporting to the AI office and also more documentation about the systems so this is the exception in the law where the technology itself is in scope whereas otherwise more the application of the AI models is in scope the law has recently been adopted and the operationalizing of it is now uh entering um into action there's hiring going on and really the setting up of this AI office that is supposed to be the enforcer of the AI law and what is unique and I think a real interesting space to watch also for researchers is the fact that the UI Act is open-ended and what that means is that it actually anticipates the known unknowns you won't hear me quoting Donald Rumsfeld often I hope it's okay here at Hoover today but we all have to appreciate the reality that there will be new breakthroughs of new AI models we just don't know exactly what those breakthroughs will be and so the question is how any law deals with these ongoing innovations that are not precisely written down in the law but that still have to be accounted for uh in regulation and so what the AI office will do is look at these new capabilities and then assess how much of a risk they entail and then what kind of mitigating measures uh are to be matched to reduce the risk and protect people this is a brief summary of the eui ACT I'm sure we'll talk about it more but uh that is one part of what I wanted to share now switching or zooming to the global level um I thought I would say a little bit about the high level AI advisory body which is an advisory body to the UN Secretary General that I have the pleasure of serving on along with 38 others from very diverse backgrounds all from all over the world but also from different professional backgrounds the idea is really to have a wide spectrum of voices in the room and I think the UN has succeeded we're only Midway in the process of our um mandate that has been um roughly one year so it's very very uh quick process we're expected to present our final report in the fall uh and have had our interim reports published right before Christmas the starting point of advice that this body gives is that we should be the UN should be complementary to other initiatives whether they are National or multilateral that are under way so we don't want to reinvent the wheel we don't want to overlap with what is working well but really build on the unique role that the United Nations has anchored in international rules uh built on the universality of Human Rights and um focusing on broader agendas uh that are continuing to be important at the UN level which is the SGS for example and we look at how AI might play a role in achieving them critically The Advisory body recommends that the global South be represented far more than it is today we see that uh people communities their lived experiences are often excluded from any policy debate even though if you look at the numbers of people impacted it's obviously the bulk bulk of the population of our world and also the unique contexts are really important to keep in mind because where are here we may be talking about uh how to regulate AI in other parts of the world the question is how do you think about AI when you don't have electricity the challenge is also how can people in the global South even have access to supercomputing how can they interact with it develop it uh and then think about their own policies in their own contexts only 35 countries in the world currently have supercomputing capabilities so there's a big gap to be filled the advice we give to the Secretary General focuses very strongly on the need to strengthen the public interest over the outsize power of tech companies in the AI space and we recommend to work to avoid policy fragmentation I'm sure we'll talk about it today but with everybody doing their own thing the risk is that we're going to have a patchwork of measures taken between the EU the US the G7 and the question is how do they all interoperate how do they work together uh in order to get the best kind of um outcomes for people so we're only mid way as I mentioned if you're interested um you can read our interim report it's actually quite modest only 20 Pages the title is slightly less modest it's called governing AI for Humanity I'm personally encouraged about where this advisory body is going I was quite skeptical at the beginning but I think the firm belief by all members that the recommendations should be anchored in international law and respect for Universal human rights is great and I personally hope that with this advice and with the UN stepping into the space of governing AI beyond what it does through its individual agencies we have the potential of creating a sort of global bottom line underneath which behavior by companies and countries cannot fall so to be short I see around the world enormous momentum in global governance um also a sense among governments not to repeat the mistakes from the past to be too late to govern this new technology as perhaps they were with social media there is a broad agreement that governance is needed and is not in conflict with Innovation but is really needed to steer it in the right direction uh and we even hear companies in the AI space saying that which is also unprecedented I believe it's less confrontational between the private sector and the public sector now of course there's on the one hand competition every jurisdiction country multilateral organization wants to have its own remit and focus on AI governance but there's also growing collaboration which I think is encouraging there is a lot of convergence around risk and safety the topic of today but there are other areas that are much less focused and I thought we would also uh focus on those a little bit because while safety is a clear priority of the UK the US and now also Canada um the the legal basis for the uh AI safety institutes in these countries is is not there in the EU it is there as I mentioned uh and the EU very much believes that it puts them in a first mover Advantage meaning laying down the new rules in a new law um may have others around the world looking to its model for inspiration and causing a ripple effect there are still a lot of questions to be answered for all of these jurisdiction itions the US the UK Canada the EU basically everyone who's dealing with AI governance meaning how will enforcement work when will oversight take place so upstream or Downstream will the enforcement be looking at the very nitty-gritty workings of the systems and the reporting on AI applications or will there be a focus on the way in which AI applications interact with people and how they spread around Society in their impact the EU emphasizes the moment that the application hits the market and interacts with people and uh I'm guessing but we'll hear more from the leaders there that the UK and the US would like to intervene before the systems are plac into the market to avoid safety risk coming from dangerous um systems released into the wild now the one thing that is missing from both the executive order in the US and the EU AI act are rules around the military uses of AI systems which I believe create risk both through the specific technological uh aspects but also through the way in which they are used the fact that the eui ACT doesn't focus on Military applications has a logical reason but it's not necessarily helpful uh EU member states are still responsible for National Security so the EU um sort of federal level is not not mandated to focus on issues of National Security but effectively that means that even if there are now harmonized rules across the EU they do not apply to the use of AI in the military and similarly uh in the executive order the um the new rules and implementations of the executive order don't speak about military uses of AI the E uh EO says explicitly does not AI when it's being used as a component of a national security system or a weapon system for example and that really leaves the two major Democratic blocks in the world without binding rules on what types of AI systems the military or intelligence Services can use for example and I think we have to ask ourselves how meaningful are the rules that have been adopted without covering the use in combat zones uh in battlefields especially as we already see that there is a lot of risk emanating from embracing AI in these context and there's also a lack of evidence that these weapon systems can comply with international law such as distinction when it comes to targets or proportionality still they're being sold around the world and the Venture capitalists around the corner here are very excited about defense Tech so my conclusion here today um on the question of global governance is not only what the EU is doing and what the UN is doing but we should also look at what they're not doing and the risks to personal safety uh to the upholding of the international order to issues of applying a mandate and oversight to the use of these very invasive systems look like and I hope we can include the use of AI by the military in our discussions today thank you that's a great start um I have the burden of introducing myself um I'm Philip Zelo I'm a senior fellow here at Hoover and I am not an AI expert U but hey I'm at Stanford University so of course I know about Ai and Technology I breathe the air I absorb it by osmosis uh I got pulled into this issue a little more than a year ago by a group of people who are working on it because they knew I had experience in how to work on policy problems including policy problems involving dangerous Technologies Wars disasters uh it actually turns out to be the third time I've been involved in a national effort working on technology problems more than 20 years ago I led an effort for the marle foundation on technology and National Security that helped lead to the creation of things like paler here in which I own no stock and then about 10 years ago I was the managing director of an effort called rework America on Tech how the digital economy might interact with American society so now I've been pulled into these AI safety issues and I've thought a lot about how to take the AI safety concerns and translate them into policy and so my comparative advantage is not my Mastery of the science of AI it uh if I have any comparative advantage at all it's because I have some experience with how to try to solve emerging public problems so what I want to talk about today is the emerging Global agenda to do that I'm going to cover the politics of AI safety briefly I want to introduce the concept of Designing safe harbors I want to talk about the Synergy of risk assessments and countermeasures and the Synergy of government and Frontier companies I know that introducing future topics with words like Synergy is almost like a promise that later on I'm going to be deadly dull and that will probably be the case but I'm going to try that it w not to be the case I think there's actually some interesting things here let me start with the politics of AI safety so there are many concerns competing for attention uh Mara did a very good job of introducing the range of concerns that the European Union is trying to address in the AI act the range of concerns that are troubling the United Nations highlevel panel there you know everything from elections to equity to opportunity to um systematic algorithmic discrimination and more so many clamoring for attention and there's a sense that if you call out certain AI safety risks maybe you're uh you're taking away the attention that should be devoted to something else that you care about I'll just tell you straight up the focus of my presentation is very specific and it's on catastrophic risk so tomorrow uh Stanford Hai is going to do a broad gauge Symposium that's going to cover the whole Waterfront of the issues of AI and Society but that is not this presentation this presentation is narrowly focused only on the stuff that can cause gigantic harm so you can Define catastrophic risk however you think best let's say something that kills thousands of people yes that would count thousands of people massive societal trauma Global tension yes that it will seem like there's a before and after the event yes so 911 scale event which killed 3,000 people cost $50 billion and relatively immediate costs and so on yes that would count and some of the catastrophic risk could be on a scale much larger than that so this is just one subset within the area of AI and catastrophic risk uh you could summarize this as dealing principally with issues of potential weaponization so weaponization right now the principal concerns in weaponization of AI have to do with biological concerns cyber maybe chemicals and then there is a set of concerns having to do with loss of control one of the things I want to call out is that the politics then of how to think about catastrophic risk or have also become somewhat poisonous so uh everyone here in the valley probably has heard of the culture War between EA and eak between effective altruism and E accelerationism in which uh there's a sense that whether uh it's a set of cultural beliefs about what companies ought to try to do and what they're about in which the companies uh some people are arguing the companies have a real obligation to try to guard progress and other people think they have a real obligation to Foster progress and they worry that these two sets of belief are in conflict and this became a bit of a dust up for instance in the battle over the governance of open AI I want to situate the take here on catastrophic risk is kind of in between the two extremes which is kind of one side you could call them the doomers and on the other side you could call them say let a thousand flowers bloom let a thousand AIS Bloom don't do anything to constrain open source models they will make the world better and better and if we need to get ready for the succession of machines taking the place of humans let's start writing our will uh so between those two extremes the concerns here have to do with a a fairly a set of bounded concerns about catastrophic risk that are already becoming evident even if they have not materialized this year how then uh should someone think about since there are these extreme views about the risk how do we sort this out my argument to you is simply that I'm in a position of grave uncertainty so let's suppose you wanted to try to figure out how how bad are the catastrophic risks oh let's say you turned to the Turing Prize winners who shared the prize in 2018 for kind of the foundations of modern deep learning well that would be let's see Jeffrey Hinton Yoshua Benjo and Yan laon okay and I would let's say we talked to Hinton Benjo and laon about what do you think are the catastrophic risks they will not agree I I think I've heard from all three of them and I they do not agree yeah so then here I am I'm not an AI expert I told you but put yourself in the position of someone who is in a situation of public responsibility as if you had public responsibility the rational answer would be I don't know but I'm worried I'm uncertain uh what are the odds I've read odds this year of a catastrophic Hazard that ranged like in the 5 to 20% range so these are non-trivial numbers and whether you you think it's 5% or 20% that's already enough well I better have a I better have a riskmanagement strategy so one thing I will tell you uh and I'm confident about is that if a catastrophic risk materialized that causes really catastrophic harm there will be enormous public action that I can tell you for sure after the covid pandemic broke out um the United States of America just at the federal level spent $5 trillion in discretionary fiscal policy that didn't necessarily save the more than a million Americans who died or the more than 20 million who died worldwide but it gives you some sense of what people do so I've LED disaster investigations before I was the executive director of the 911 Commission and then I led a national investigation into the covid crisis that used the book lessons from the covid war I was the director of the covid crisis group and I want to shout out my Stanford colleague in that group David rman who is an outstanding virologist I will tell you one takeaway from these investigations is if we're going to do gigantic stuff to manage catastrophic risk big takeaway deep Insight before is better so one thing you can do though if you think about the uncertainty is you can say not you're hedging you could adopt the following assumptions you could say Innovation will happen you could say it will proliferate uh including people stealing software you could say serious risks are both plausible and their and their likelihood is uncertain and you can say that therefore these risks are at a level that have to be mitigated and countered you have to prepare to mitigate or counter them and then uh doing it before is better so that's politics of AI safety now let me turn to the idea of Designing safe harbors we have a number of historical models for how to gain insight into dangerous Technologies and try to control them and all these analogies have been suggested in the AI setting and it's worth briefly noting them there's the nuclear analogy um for instance we set up govern in the nuclear case we set up governmental labs to try to gain Insight after we had the atomic bom casualty commission right after hirosima and some of these labs are not too far from us and then on the control side what we did with nuclear is basically a trade in which people accept a nuclear hierarchy and very rigorous controls in exchange for which they can access scarce materials and high technology and we institutionalize this through things like the restricted data system for information control and the international atomic energy agency now notice that this kind of trade is very uh will arouse a lot of concerns politically because a lot of people don't accept politically that they should accept an hierarchy in which the United States is once more sitting at the Apex deciding who gets in the door and there's also a big argument as to technologically whether or not you uh can control the access including the issue of Open Source systems so the nuclear analogy is actually politically somewhat toxic to a lot of people and technologically contestable you have the biological model there again we set up Labs but actually here the labs were a mix of both government and private Labs not in the nuclear case they're overwhelmingly governmental through much of their early history here it's a mi more of a mix of government and private and by the way very lightly regulated uh one of the efforts control was the biological weapons convention ratified in 1972 and another effort at sharing and a little bit of control was the World Health Organization these entities tended to operate by consensus they believed in sharing and their enforcement tools were weak the result then of that system has been basically nothing done about the inequity and access to Advanced biology a gigantic bi an industrial biological weapons program created illegally in the Soviet Union in the 1980s and discovered only at the beginning of the 1990s and the outbreak of a global pandemic which these organizations did nothing to prevent or contain so that track record is not necessarily encouraging there's the greenhouse gases where we have the ipcc system now what this does is it organizes Global Science to at least share knowledge build consensus on the concerns and established and establish possible Norms it works consensually and does not seriously attempt control though there are efforts to pass conventions and enforce the rules of those conventions and you can see how well that's going but I think the information side of the ipcc has been very powerful and is an interesting analys ology and actually that's already been adopted in the AI case then there are regulatory approaches like lure in the uh in the case of the EU for example you can think of the lure approach which is I'm just going to create a Gateway and people are just going to have to get a license to operate you can think about lure for the use cases which Mara referred to and which is where a lot of the EU a Act is focused or you can think about lure for the models themselves and they haven't really fully gone there yet and that's being contemplated uh you can really imagine uh quite elaborate blueprints for try to set up comprehensive regulatory lenser schemes in my view one of the best efforts to imagine such a blueprint is this report that came out in February from Gladstone AI I think they did a good job there are things in it I don't agree with but I think it's a a substantial respectable effort worth people's attention what I want to call a little bit more attention to though is what exists already which is liability if you build dangerous products you're liable for what happens to a very large degree and the more dangerous the products the more liable you are for what happens even if they are being used in unforeseeable ways and they're being misused I think a lot of people in Silicon Valley are used to thinking that new technological developments have a safe harbor because they grew up under Section 230 of the communications act that kind of sheltered platforms I just want to tell you that Safe Harbor is not available for AI models and I think this re this realization is Dawning on a lot of people now you can read this headline this is in the wall Street Journal on March 29th so 17 days ago and there is a significant discussion in this article about the catastrophic risk and by the way the liability point is relatively simple if I hire basically if I create or hire an AI agent to do something for me I've got a principal agent relationship the same way I would if I hired a human to do that if hiring the human to do that like say to carry out Mass deception would get me in trouble it's not like you avoid that liability by hiring a computer to do it but then if you create a product that is abnormally dangerous that's a term of Art and tort law by the way um the level of liability kind of the way they Define negligence there used to be a Doctrine called strict liability um tort law professors argue over whether it's really strict liability or just a really high standard for negligence in which if it's abnormally dangerous you should foresee misuse and um you should foresee the unforeseen and you'll be held liable for that um it's not very well known but the only reason we have nuclear power nuclear energy in the United States for example is because the nuclear energy industry companies like Westinghouse in the 1950s said we cannot go into this business unless you pass a federal law that immunizes us from liability and actually creates a an insurance plan otherwise there will be no nuclear power reactors in America and Congress then passed such a law during the Eisenhower Administration called the price Anderson act which is the reason there is a nuclear power industry at all because otherwise the liability issues were simply impossible insurance companies could not cover it so this is the liability world is very interesting it is probably the most effective world it is decentralized and it effectively governs safety in everything around you this building and the building codes the fire codes the codes governing the use of hazardous things like natural gas electricity all this is covered in without any regulation being issued by a federal agency at all it's basically the the threat of liability causing uh uh the insurance industry to create public private entities usual involving Engineers usually which you've never heard of that write constantly evolving technical standards for how to use things which are then codified in all kinds of codes that mostly you're unaware of unless you're a homeowner or you have to build something but are by the way extremely effective and are used now worldwide so the liability area is actually very interesting and then what happens then is if you build to code you've basically got a safe harbor as we did more and more work on how to make buildings safe from fire it essentially became best practices the best practices become more and more codified they become the Safe Harbor for Builders so you may think though that idea of a safe harbor might be appealing it might be appealing for Frontier companies it might be appealing for the government you might then say well what would you need to have in order to have a safe harbor here's my version of the requirements for a safe harbor First Independent evaluation it's not good enough for the company to say we invested in evaluation we're good we're a conscientious company and by the way I think some of them are and some of them are spending serious amounts of money on um what is they often call alignment but which is more than just alignment but in if you want Trust it's got to be independent evaluation the evaluation has to be well regarded by authorities and authority figures are both public and private actually the evaluation has to be both pre and post- deployment this is a big idea by the way if you want to have independent evaluation pre- deployment that's a big idea and requires some very delicate discussions having to do with IP in a superly fiercely competitive environment if it's done well you will then get a a better and better idea of what best practices look like which will then turn into the standards of care that can shelter people from liability incidentally if you think that the Civil liability for companies making these products isn't clear enough yet government can foot Stomp the liability dangers without any legislation authority figures in government could emphasize liability and call out the existence of better standards and best practices in ways that would be wonderful evidence in any plaintiff lawsuit you can create an environment without legislation in which standards of care are already publicly evident to the company or you could argue they were and then that will have an effect in they the way they would assess their risks okay the last thing I want to call out is that we are already creating emerging institutions for insight and assessment so Mara is already a member of the UN highlevel panel that's important here at Stanford you have things like an AI index report that Stanford hii is putting out I want to really call attention to um the ipcc model is already being endorsed and created in the form of an international scientific report on Advanced AI safety the first such report will be later will be next month at the next AI safety Summit in Soul Mara and I were both at the first summit that was held last November at Bletchley Park and they have an expert advisory panel that will produce that report it's chaired by Yeshua Benjo you will hear from Benjo in the next session who will give you a little bit of a preview of that report so that model the kind of the global sharing of information help build consensus maybe established Norms people are already moving out on that the other big deal is the creation of the AI safety institutes you will hear later today from the heads of the two the two existing ones the Pioneer were the British the leading team for the British are all here you'll hear from them this afternoon the Americans are right there with them you'll hear from the director of the American Institute in the next session um the British right now have more people and more money and I want to call out two things about the AI safety institutes that have been created so far first they are not regulatory they are designed to gather information to reduce uncertainty about the risk and conduct evaluations the determination of what to do about what they find is not in those entities and in my view it should not be in those entities their job is to produce information about risks they are not constituted to judge what risks Society should accept we'll have Democratic institutions that will judge what kinds of Institutions Society what kind of risk Society wants to accept in relation to the tradeoffs that's not necessarily their job their job is to inform those judgments so not necessarily Regulatory and by the way not being regulatory is the prerequisite to gaining the trust of the companies whom you're going to have to work with intimately to do the work there are precedents for this by the way in other Industries the second thing I want to call out about these institutions is they're they have the chance of being multinational to their enormous credit the British and the Americans two weeks ago announced that they were going to pull their efforts they've created the first AI safety institutes they signed anou they announced two weeks ago in which they said we're going to pull our work and pool our efforts which then means that if necessary the pulled effort can present a face to American companies looking to an American Authority or an American entity and also say to a British entity so they the same entity can work say with Google Deep Mind on the one hand and open AI on the other and actually Google's in both sides they're all on both sides of the Atlantic multinational safety institutes are going to be important what will also be important though is that the companies are not going to want to work with a 100 intermediaries so it's going to be important for the intermediaries to combine their forces let me turn to the Synergy of Assessments and countermeasures is a point I think is not well understood the people who do the evals are going to be the people you're going to lean on to help design the counter measures for bad AI because they're going to understand they're the the Insiders will have the best understanding of how things could go bad now add to that who actually will do the international risk assessment so when you think of international risk assessment don't think of which you know don't think of analyzing what m trol is doing think of analyzing what Russia is doing in Ai and North Korea and others when I think about the Russian president I think about this which I remember firsthand this is a really bad story for those of you who never learned about it it was entirely clandestine and it was massive it got discovered through British intelligence then combining with American intelligence in 1989 and 1990 and then got cleaned up substantially but not entirely during the 19 90s due to some extraordinary efforts and is not very well known but it gives you a sense of man everything is possible so as a government you have to then prepare when you're doing the international risk assessment you have to then say we don't have to just evaluate what the companies are planning to do we have to evaluate what the worst people in the world can do empowered with high technology that may mean you have to develop danger assessments of dangers and develop models reflecting those dangers at levels well beyond where the companies want to go for good reason so then think about that for a moment that means that assessing the greatest dangers requires doing highly dangerous research then think about who is doing that highly dangerous research and doing it under what security conditions with what relationships to the necessary government agencies and you begin to see what I mean about the Synergy here which is a very interesting and difficult problem one thing for sure is that the government itself simply to conduct the dangerous research will have to formulate Safety and Security standards to govern its own work which it can do Under the defense production Act without any new legislation which will then necessarily say set best practices and safety standards for any company work that might be nearly at that level so what you see here for example on the left that's a schematic for a bsl4 lab that's bios safety level four lab when we do dangerous research on pathogens which we do now which is a big deal bsl4 Labs there are about nearly 60 of them worldwide are very difficult to build and very costly to maintain what you see on the right is a picture of Stanford lab which is not at bsl4 by the way it's at bsl3 which is a lower level but it gives you some sense of what happens in these Labs so the biological analogy is really interesting because the biology example can tell you that it's possible that the best defense to bad AI may be good AI for example if you were trying to defend against the worst possible disease or pathogen this is a World Health Organization standard chart and what you would learn from this chart is that if I want to build a vaccine I have to build the pathogen I have to understand the worst AI in order to develop the countermeasure for it that's what we do in B that's why these are bsl4 Labs because bsl4 labs are building and working with the most dangerous pathogens on Earth because that's the only way to develop vaccines for them and the same thing may be true in the AI context final point the Synergy of government and Frontier companies notice that the frontier companies need safe harbors while they also need the government to protect their I because the safe harbors require predeployment testing and independent evaluation of that if you're going to do it in a way that invites trust the governments however need the the companies because only the companies will have the capabilities to do the kind of Assessments and counter measures that I'm talking about even though the companies themselves may not want to go there for their private reasons government doesn't have the compute government doesn't have the data centers government doesn't have the access to electrical power even that the companies will need to do their training runs so government if it wants to do work at on these problems at this level is going to need compan resources to help them do it and I don't think we've even begun to really work through how to design that and and manage that relationship so here's what I want to then call out is I want to call out the possible role of the safety institutes as trusted intermediaries if the safety institutes are not The Regulators but they're deeply involved in the companies in doing the predeployment stuff the safety intitute will also become in effect the labs that have unusual insight into the dangers and therefore also into how to design the counter measures the safety institutes then may become simply crucial in playing the role of the trusted intermediaries between the public and private world that will arm whatever we decide to do in public policy which is one of the reasons I'm glad you're hearing from the Pioneer leaders of the safety institutes later today so let me stop there and let's open the floor in our remaining time to uh to your questions yeah what was uh the biological program the Soviet Union that you mentioned which is so scary what could have been done to prevent this or what did we learn um we should have done and and maybe how can we translate this to AI today great Point um we had a treaty that they simply violate comprehensively on an enormous scale involving thousands of people with industrial complexes and places like Kazakhstan so the treaty wasn't good enough we had no particular enforcement mechanism behind the treaty we began to get evidence that the Soviets that something might be a miss when there was a suspicious outbreak of Anthrax in what was then called sperl which is now a cinberg again um in 1979 the biology Community by the way poo pooed those reports and and uh in general the biology Community believed all through the 1980s that people who were getting hysterical about those reports were uh were reaganite conservatives and Uber Hawks and that actually there was really uh nothing much to worry about there they took the natural outbreak explanation is probably being true um and then we started getting the defectors from the program I will say that when the first Defector came over which was in O October 1989 to British intelligence who then shared this with the United States government this was incredibly secret between those two governments none of our allies knew about it for some time thereafter uh in part because we thought it would blow up the whole diplomacy to end the Cold War if this came out and we wanted to have very quiet discussions about this with M gorbachov and the others so that kind of lessons learned from that well one lesson is um I will tell you that the int when the intelligence people found out what was going on it was actually much worse than they thought was possible they thought there might be in the worst case maybe they are running an experimental program they had no idea that there was an industrial scale program like this so it's the first thing is it it induces humility and then invites you to think the second thing is you need to take your intelligence Mission really seriously so under the BWC we had a treaty we had very little we had very modest intelligence work to track this sort of thing around the world we had very limited National Technical means to try to monitor this sort of thing and made very little effort at it and then when we got some of these reports we tended we turned to the biology community and since they discounted them our government's discounted them too except for a hand handful of Heretics and so it's a very sobering experience therefore the um one of the lessons of this for AI is you have to take the international risk assessments really seriously and that that therefore has these grave implications for the way we organize our work I'll say that right now our government and our intelligence Community is way behind on this stuff is um there is more there's more work being done on AI safety in private companies right now than in the American intelligence Community I don't think it's even close so th those are some of the lessons I would call out well I won't pretend to be an expert on bioweapons in the Soviet era but I do think this question of a loss of control and who has agency over understanding risk is very important so in the AI safety discussion there's a lot of um debate about you know when might people Engineers lose control over AI systems but if you zoom out a little bit I think we can conclude and you basically ended on that note that governments don't have the control or oversight now over what is happening inside companies and I think that lack of insight means that there's not a good public discussion about what public policy should look like and how we all must understand risk not just a few people in an agency who you know May well gain the capacities to do so not just a few people inside companies who actually themselves admit that they don't have a handle on how models are going to behave so maybe we have to conclude that we're in a very dangerous situation where nobody has a real sense of control over the role that AI is allowed to play in our societies and I wanted to pick up on a point that you made in your presentation that of safe harbors and you mentioned um section 230 so there may not be explicit safe harbors for AI developers now that exempt them from liability but of course there are implicit safe harbors in that there's just a lack of law that governs not only the AI models where we are now but also what went before and I think the outsize power of big companies the role of big Tech that has had so much space in this country cannot be discarded because what is a core ingredient of AI models data companies were allowed to collect process scrape data from the internet without any legal coverage not an explicit Safe Harbor but also not an explicit prohibition to do so and so the power that they amassed in that process was then the sort of stepping stone for the power that they have now in developing these models and so I do think it's really important to to look look at the notion of control inside and oversight in a much more comprehensive way than just trying to isolate what any given AI company or AI model is capable of we have to see it as an ecosystem uh and I think that that would be um much more useful next question um I'm actually from London so very interested to hear the the interaction with the EU and and and the US um my question is when you're when you're trying to make policy decisions with in complete information and you have predictive AI which you can use statistical tests for and generative AI which is it's kind of like grading a math test versus a history essay has any thought been given to experience from drug discovery not AI to pick a compound but on making decisions with huge consequences life or death in compete information with with uh policy considerations back and forth has anyone thought about using some of the lessons from that to inform and that was making decisions under conditions of uncertainty where where the consequences of getting it wrong are literally human life uh there is a whole um literature in in engineering world for example about uh risk management um and how to manage uncertainty I have not I've kind of lived these problems more than I have studied them formally I'm aware of the literature on this let's say in in intelligence and foreign policy decision making um I don't think in in general it's a very good literature um the um the main concern here is that you take should take away from that literature is you should not try to handicap what will happen your job is to handicap the most plausible possibilities of what could happen and the threshold of plausibility is this is worth time and money so a comparison that I've heard um and I think we can draw from with the whole medical field or Pharmaceuticals are a couple one I think it's an important lesson to take away that even very complex challenges can be tackled and it's not a one-size fits-all but when you look at the field of medicine there are interventions and requirements at almost every step of the way the way the drugs are produced the way they're distributed through the pharmacies the licenses that the doctors need in order to prescribe the um information that you get as a patient you know all to to try to prevent harm essentially so I think that's one thing the the other analogy that I've heard and I think it can be quite useful is the question of this does relate to bio engineering research when is a safe moment to release an experimental treatment or an experimental process or research process into the market or society and with AI there are hardly any guardrails right what what is the big driving force are Market competitions companies that want to be faster than the other but not so much a controlled process of saying hey we have this very very potent new innovation we're going to research it in a lab for you know x amount of time until it is considered safe enough or to meet certain criteria to then be released and so this notion of clinical trials if you want to use a medical uh term has been used in comparison with AI to say maybe we should be more deliberate about the moment when an AI model goes from the research phase that we could potentially all benefit from gaining greater understanding onto the market instead of it becoming one major societal experiment when these models are just thrown onto the markets only driven by market forces but not by harm prevention considerations so um the binary here is the precautionary principle so the precautionary principle would argue we don't release it till we know it has no risk um and the argument is the precautionary principle I've heard andreon talked about this for example and he ridicules it with some cause about well yes in that case we would never have had fire um or you know anything automobiles right because all these risks so you've got an extreme version of precautionary principle which is an instinct and people by the way tend to be highly risk averse psychologically and oriented to that and you can scare people and then you've uh got an argument which you know you gets picked up in the culture War here in the valley of you know we need to Foster acceleration so the middle ground that I've tried to find and some people I work with try to find is innovation's going to happen whether uh and if if we don't innovate that may we may find ourselves in an even worse position so innovation's going to happen we have to figure out how to manage the risk and we may need to deploy things that have certain risks and accept certain risks and accept that as a society we need to manage those risks because we're doing cost benefit calculations in an environment of uncertainty I do think that uh therefore we want to understand the risks as best we can and understand how to manage them in part if if you understand a little bit about how to manage the risks people are much more likely to accept them and in Foster Innovation rather than be scared and paralyzed that gentleman in the blue jacket very good so thank you very interesting discussion so question is about so I'm from Germany originally so Europe United States liability you pointed out the concept of liability at the very beginning liability is dealt with very differently in Europe versus United States perfect example recently a bunch of companies decided to use the software you know applied it to a bunch of millions of cars right and as of today the liability United States was dealt with like you have to buy the car back and you get be reimbursed in Germany it's not a single person sitting in jail yet so how do you want to apply liability you know as a concept to something as complex as II so let me try to draw from your example which I think is in part an enforcement problem I mean um when executives are deceptive about products or when there's a harm discovered in any product then of course there should be consequences uh and it you know there will obviously be differences between um the EU and the US and how that happened but the fact that there is nobody held accountable for I think you're talking about the Volkswagen emissions um software Scandal is also a problem of enforcement and perhaps points to the power of the car lob lobbying industry as well but that's for another discussion um I think on the one end it is important and that's probably also where um Philip and I disagree um on whether it's better to have non-binding oversight or oversight on the basis of binding law when you have binding law including on liability then the possible consequences when when the laws enforced well are obviously firmer um Philip mentioned that companies don't like to have to adhere to different um standards which is true they prefer a Level Playing Field but if they have to choose between adhering to a law that's binding or a law that's non-binding they're probably going to go for adherence to the law that's binding because the consequences of non compliance are Graver so when you think about liability or accountability of companies in whichever field I think is a combination of factors of how forceful is the law and how well is it enforced what kind of sanctions apply uh when there's violation and we're not even close to seeing seeing that sort of mechanism mature in the case of AI I think we're seeing baby steps taken towards how we can build such a mechanism with different political choices being made let's be fair about it you know sometimes by by um lack of options I think in the United States a reference to existing law is also because there are zero expectations of what Congress might do legislatively so what can you do use executive power and use existing laws it's a different situation in the EU how will that play out we'll have to see but we now have different models to look at uh and in in both models or all models enforcement will be key and we don't talk about enforcement enough this gentleman the I'd like to expand a bit in the medical field uh it's not just risk and benefit the thing missing is informed consent and I think AI has to address that the developer of the product not only has to demonstrate that it's effective and establishes safety but inform the user and I think we have to discuss that issue I'm I'm not going to argue I think probably it'll depend on the context of the use case and that'll be our last one for this session oh hey um so my question is around the impact of potential geopolitics um I remember reading in the coming Wave book by Mustafa salaman that one of the challenges is like even if say uh countries like us or European union and others like come together how do we handle countries like the example you mentioned about Soviet biological weapons right like how do you how do you deal with countries which may not comply with any of these regulations you prepare to protect yourself I mean it's a real problem um and I gave the biology example of the aspiration to universality and then the way they approach the in enforcement and surveillance and so forth and this is this model has not worked well um so then one of the responses for instance in the biology case in the pandemic prevention case is you have to supplement you know voluntary guidelines with uh serious International intelligence efforts to do Global biological surveillance for disease outbreaks without Reliance on Country reporting for example which by the way we weren't doing in a in a sufficiently serious way before 2019 so in this case you you have you have to assume bad things you have to it would be nice if you didn't have to but as a person if you were in a position of public responsibility you have to assume pretty good chance of a divided World pretty good chance of some non-compliant Outlaw governments that have some technical skill so then you just have to prepare accordingly and uh but I think actually um we're we're and we're building institutions that are G that that we are giving us the possibility of developing the capabilities we'll need as they mature and as we add on further refinements um we're in a race now between policy and the development of the technology government is way behind think n would agree on that and then the question is whether or not um government will catch up and if so how do we design the governance role in a way that's um that's going to work um and you saw my argument is that government may actually need to Foster Innovation if only to understand the dangers so some people take the example that you sketch that of global competition and what if you you restrict your own sector uh what will happen to your opponents as an argument not to do any kind of oversight in the United States I think it's a familiar argument we will probably hear a little bit more about it later today as well um and I think any Improvement whether from a moral perspective or from an actual enforcement perspective has to start somewhere but what is unique with AI is that so many people are convinced that there is risk in the deployment and behavior of these models and so if you believe that then the question is do you disadvantage yourself by putting guard rails on or does the jurisdiction that does not put guard rules on invite the biggest risk upon themselves by having unrestricted use of these systems so I think just from a sort of logical consequence of the argument and from what is morally correct to do uh having boundaries that that mirror the very principles existing laws Notions of non-discrimination protection of the Constitution very important um anchors of a free Society Merit having guardrails even if you don't know exactly what your opponent is going to do with the with the reality that that creates all right a quick Break um re being back here actually in about five minutes [Music]

Channel: Hoover Institution

Views: 1,065

Keywords: Challenge of AI Safety, AI safety and catastrophic risk, risk assessments, World’s First AI Safety Institute, Reid Hoffman, Eric Schmidt

Id: TAd3SxLO8Ng

Length: 70min 30sec (4230 seconds)

Published: Thu Apr 25 2024