The AI Alignment Debate: Can We Develop Truly Beneficial AI? (HQ version)

okay well in which case let's crack on so uh ladies and gentlemen get ready to meet the cunning Maverick of Silicon Valley the one and only George hotz renowned for his daring exploits Hots commands in enigmatic Persona which merges the technical finesse of Elon Musk and the wit of Tony Stark and the charm of a True Tech Outlaw now many of you would have or indeed should have seen this man on Lex's podcast recently for the third time no less from crafterley jailbreaking the supposedly Invincible iPhone to outsmarting the mighty PlayStation 3 he's proven that no Tech Fortress Is impregnable once targeted for his audacious creativity by Sony with a lawsuit this hacker wizard stoically dance past the curveballs thrown by the tech Giants all achieved with the graceful swag of a street smart Prodigy now when he's not out foxing major corporations you'll find him at the heart of the Avant guard of AI technology gallantly trailblazing through the Wilds of the tech Frontier he's currently building a startup called micrograd which is building super fast AI running on Modern Hardware and truly he's the James Bond of Silicon Valley minus the martinis of course now uh please welcome the unparalleled code Cowboy the Unapologetic technomancer George hotz whoo anyway also joining us for the big fight this evening is the steadfast Sentinel of AI safety Connolly he undeterred by the sheer complexity of artificial intelligence Conor Braves the cryptic operations of text generating models with Steely resolve now about two years ago Connor took on the Herculean task of safeguarding humanity from a potential AI apocalypse his spirit is Relentless his intellect razor sharp and his will to protect is unwavering now drawing on his contentious claim that we are super super [ __ ] uh yeah uh Connor channels the urgency of our predicament into his work now his startup conjecture isn't just a glorified Tech Endeavor but it's a Lifeboat for us all racing against the Breakneck speed of AI advancement with the fates of Nations possibly at stake he's determined to break the damning prophecy and render us super super saved so um brace for a showdown as kanalihi the Maverick defender of ai's boundaries strides into the ring now the man who declared we're super super [ __ ] is here to prove just how super super not [ __ ] we could be if we make the right decisions today so please give it up for Mr Connor super super Leahy woo now um Connor I'd appreciate it if you don't go down in the fourth I want this fight to go the distance now we're running for 90 minutes this evening there'll be a 10 minute openers from uh from uh we said Hots didn't we from from Hots first and then Connor and uh I'll only step into the ring if the punch-up gets too out of hand and unfortunately we won't be taking live questions today because we want to maximize the Carnage on the battlefield George hotz your opening statements please um yeah we're super super [ __ ] I think I agree with you well that was a short fight yeah look I think okay um so to to make my opening statement clear and why maybe it doesn't make that much sense for me to go first I think that the trajectory of all of this was somewhat inevitable right so you have uh humans over time and you can look at a 1980 human and a 2020 human they look pretty similar right Ronald Reagan Joe Biden and that's all to say um whereas a 1980 computer is like an apple II and a 2020 computer is a is a M1 Max MacBook like lines looking like this right so you have one line like this one line like this these lines eventually cross and I don't see any reason that Line's gonna stop right I've seen a few of the other guests argue something like well llms can't Problem Solver but it doesn't matter like if this one can't the next one will whatever you call I don't believe that there's a step function I don't believe that like oh now it's conscious so now it's intelligent I think it's all on a gradient um and I think this gradient will continue to go up will approach human level and pass human level now this belief that we are uniquely [ __ ] because of this the amount of power in the world is about to increase right when you think about power and you think about shut up you can just talk about energy usage the amount of energy usage in the world is going to go up right the amount of intelligence in the world is going to go up we may be able to do some things to slow it down or speed it up based on political decisions but it doesn't matter the trajectory is up or major catastrophe right the only way it goes down is through War nuclear Annihilation bio Annihilation meteor impact some kind of major annihilation what we can control and what I think is super important we control is what the distribution of that new power looks like um I am not afraid of super intelligence I am not afraid to live in a world among super intelligences I am afraid if a single person or a small group of people has a super intelligence and I do not and this is where we get to check in that a chicken man is the man who owns the chicken farm uh there's many chickens in the chicken farm and there is one chicken pan it is unquestionable that chicken man rules and if you believe Chicken Man rules because of his size uh I uh invite you to look at cow man who also rules the cows and the cows are much larger than him Chicken Man rules because of his intelligence this is basic less wrong stuff everyone kind of knows this how'd the squishy things take over the world look I agree with Elio's yutkowski all up to Nuke the data centers right um so uh I do not want to be a chicken uh and if people decide they are going to restrict open source AI or make sure I can't get access to the compute and only trusted people like chicken man get access to the compute well [ __ ] man I'm the chicken and uh yeah I don't want to be the chicken uh so I think that's my are we [ __ ] maybe um I agree that that intelligence is very dangerous how can you look at intelligence and not say it's very dangerous right intelligence is somehow safe but things like nuclear bombs are an extremely false equivalency because what does a nuclear bomb do besides blow up and kill people intelligence has the potential to make us live forever intelligence has the potential to let us colonize the Galaxy intelligence has the potential to meet God nuclear bombs do not they just blow up um so I think the question and like you have things like crypto which are a clear advantage to the defender at least today and you have things like nuclear bombs which are a clear advantage to the attacker AI it's unclear I think the best defense against an AI trying to manipulate me and that's what I'm really worried about future psyops you know we're already seeing it today with the voice changer stuff like you're never going to know who's human the world's about to get crazy um the best defense I could possibly have is an AI in my room being like don't worry I got you it's you and me we're on a team we're aligned I'm not worried about alignment as a technical challenge I'm worried about alignment as a political challenge Google doesn't like me open AI doesn't like me but me and my computer you know we like each other we're aligned and we're standing against the world that has always since the beginning of History maximally been trying to screw you over right intelligence people think that one super intelligence is going to come and be unaligned against humanity all of humanity is all aligned against each other I mean we have some common values but really come on everyone's trying to scam everybody the only reason you really team up with someone else is like hey man what if we team up and scam them right hey what if we team up call ourselves America and we we uh we built a big army and say we're free and independent yeah right it's that Force that has made Humanity cooperate Humanity by default is very unaligned and has every kind of belief Under the Sun so I'm not worried about AI showing up with a new belief Under the Sun I'm not worried about the amount of intelligence increasing I'm worried about a few entities that are unaligned with me acquiring god-like powers and using them to exploit me that's my opening statement cool yeah thanks that's uh that's I mean yeah I also kind of agree with you and most of the things you say it's a few details I'd like to dig into there but for most of the things you say I do think I agree with you here I think it's absolutely let me just like start with saying I totally agree with you that misuse and like you know Bad actors were using AGI is a horrible dangerous outcome that's that's like you know sometimes the the uh less wrong you know crowd likes to talk about X risk but also sometimes I've talked about s risk suffering risks so things are worse than death I believe that you can probably almost only get S risks from misuse I don't think you can get S risks probably like you you can but it's extremely unlikely to get it from like just like raw misalignments like you'd have to like get extraordinarily unlucky so while I do it so I do think for example a very you know controllable AGI or super intelligence in the hand of sadistic psychopath is significantly in a sense worse than a paperclip maximizer so I think this is something we would agree on probably sure so I I think I'm I think I'm pretty much I'm bored with you on a lot of things there where I think thing things come about a bit of the tale as I think there's two points where I would like to take as my opening statement two things one I want to talk about the first one is I want to talk about the technical problem of alignment so am I concerned about the kinds of things like misuse and like small groups of people centralizing power potentially foreign yeah I I think this is a very very significant problem that I do think about a lot and that'll be the second thing I want to talk about the first thing I want to talk about is that I don't even think we're going to make it to that point I don't think we're going to get to the point where anyone as a super intelligence that's helping them out we're good if we don't solve very hard technical problems which are currently not on track to being solved by default you don't get a bunch of you know super intelligence and boxes working with a bunch of humans you get a bunch of super intelligence you know fighting each other working with each other and just ignoring humans humans just get cut out entirely from the process and even then you know Pro it's you know whether one takes over or they find nicolibia I don't know like you know who knows what happens now but by default I wouldn't expect humans to be part of the equilibrium anymore once you're once you're the chicken man well why do you need chickens you know if you know maybe if they provide some resource for you the reason humans have chickens is that they make chicken breasts I mean personally I wouldn't like to be harvested for chicken breasts just my personal opinion I consider this a pretty bad outcome um but even then well as a chicken man finds a better way of chicken breasts or you know modifies himself to no longer need food I expect the chickens are not going to be around for much longer you know once we stop using horses for transmutation didn't go very well for the horses so that's kind of the first part of my my point that I'd like to you know maybe hear your opinions on hear your thoughts on is that I think the technical problem control is actually very hard and and I think it's unsolvable by any means I think like you know you know you and like you know a bunch of other smart people work on this for like 10 years I think you can solve it but it's not easy and it has to actually happen and there is a deadline for this the second point I want to bring up is kind of where you talk about how humans are unaligned I think this is partially definitely true I think I'm unusually I am the more optimistic of the two of us in this scenario not not as not a role I often have in these discussions where I actually think the amount of coordination that exists between Humanity especially in the modern world is actually astounding every single time two adult human males meet and don't kill each other is a miracle have you seen what happens when two adult male chimps from two different War bands meet each other it doesn't go very well and those are already pretty well coordinated animals because they can have warbands what happens when you know two male bugs or you know I don't know sea slugs meet each other you know either they ignore each other or you know things go very poorly this is the default outcome the true underlined outcome the true default state of nature is you can't have two adult males in the same room at any time I I saw this funny video on on Twitter the other day where it was like you know some Parliament I think in east Europe or something and there's this big guy and she's like going at this politician who's like in his face he's like screaming he was like going everywhere and not a single punch will throw him no did no one took on a knife no one took out a gun and I was like and I was like wow the fact that we're so civilized and we're so aligned to each other that we can have something this barbaric happen and no one throws a punch is actually shocking this is very unusual even for humans if you go back 200 years punches and probably gunshots would have flown so this is not to say that humans have some inherent special Essence that we're good that we have solved goodness or by any means what I'm saying is the way I like to think about it is that coordination is technology is a technology you can improve upon it is you can develop new methods of coordination you can develop new structures new institutions new systems and I think it's very tempting for us living in this modern world to it's kind of like a fish and water effect we forget how much of our life you know a lot of our life is built on you know atoms on you know physical technology a lot of it's built on digital technology but a lot of it is on social technology and it when I look at how you know how does the world go well like you know should it be only the you know special Elites get control of the AI I'm like well that's not really how I think about it I think about it way more is what is a coordination mechanism where we can create a coordination selling point or we can create a group an institution a system of some kind that where people will you know have game theoretic incentives to cooperate on the system the results in something that is net positive for everyone because the truth is is that positive some games do exist and they're actually very profitable and they're very good and I think if we can turn you know you can turn any positive sum game to an into a zero or a negative something pretty easily it's much easier to destroy than is to create but I think it's absolutely possible to create coordination technology around Ai and to build coordination mechanisms that are not positive for everyone involved so those would be like my two points happy to dig into any ones you you think would be it'll lead to an interesting Direction sure uh so I'll start with two and then go to one uh so two uh I moved to Berkeley in 2014 and I threw myself the Mary cult uh I showed up at the Mary office and I'm like hi I'm here to join your cult uh it's a true story and what I started to realize was Miri and less wrong in general have a very poor grip on the practicalities of politics very much I think there was sort of a split uh you know uh uh Curtis yarvin like new reaction this is a spin-off of rationality and it's a spin-off rationality that understood the truth about human nature so when I give you that you give that example of two chimps meeting in the woods and they're gonna fight if I'm one of those chimps at least I stand a chance right he might beat my ass I might beat his but if I come up against the FBI things do not look good for me in fact things so much do not look good for me there's no way I'm gonna beat the FBI the modern forces are so powerful that this is not a oh we've established a nice Cooperative shelling point this is a we have pounded so much fear into these people that they would never even think of throwing a puncher firing a gun we have made everybody terrified and this isn't good we didn't we didn't achieve this through some enlightened cooperation we achieved this through a massive propaganda effort right it's the joke about you know the American Soldier goes over to Russia and it's like uh man you guys got some real propaganda here and that the Russian soldiers like yeah no I know it's bad but it's not as bad as yours and the American Soldier is like what propaganda and the Russian just laughs right so so this this didn't occur because of um this occurred because of a absolute tyrannical Force decided to dominate everybody right um now oh I think so uh I think there's a way out of this I think there actually is a way out of this right and I wrote a blog post about this called individual sovereignty and I think a really nice world would be if all the stuff to live food water Health Care electricity We're generatable Off the Grid in a way that you are individually Sovereign and this comes back to my point about offense and defense right if I have a world where you don't want it to be extreme defense you don't want every person to be able to completely insulate them but you want like okay it takes a whole bunch of other people to gang up to take that guy out right like that's that's a good that's a good balance um and the balance that we live in today is there is one pretty much a unipolar world I mean thank God for China but um you know there's one there's one unipolar world we've got the America and what are you gonna run I'll pay taxes I don't care if you've overseas right um so yeah my point about the coordination is that if you're okay with solving coordination Problems by using a single a Singleton super intelligent AI to to to make everybody cower in fear and tyrannize the future sure you'll get coordination yeah that works that works I'm the only guy with a gun and I got 10. I gotta name it all 10 of you and you can all die or listen to me your choice so I'm I'm curious about uh so I understand what you're saying and I think you make some decent points but um I think I view the world a bit differently from you and I'd like to like dig into that a little bit so like who do you think is less afraid someone living just a median person living in the United States of America or the median person living in Somalia sure America that's a fair well that's kind of strange Somalia doesn't have a government they have much less tyranny you're much more you can just buy a rocket launch rooms as I live in a farm and just like you know kill your neighbors and No One's Gonna Stop you so like how does that interact with your whole team those who will trade liberty for safety deserve neither that sorry I don't understand could you elaborate a bit more um in Somalia you have a chance in America you do not right I am okay I would rather live in fear I would rather be worried about someone shooting a rocket launcher at me than to have an absolutely tyrannical government just you know just just like like a managerial class I'm not saying by the way I agree with you that these things are possible I agree with you that the less wrong notion of politics is possible I would love to live in these sort of Worlds but we don't the the the Practical reality of politics is so much more brutal and it just comes from a straight up instinct to dominate not an instinct uh you know government by the people for the people it's branding yeah I mean yeah so to be clear I very much do not agree with les Ron's views in politics and a bit of an outcast for how I view how conflict theory I view politics but this is I feel like you're kind of dodging the question here just a little bit it's like well if that's true why aren't you living in Somalia I know people who've done it right it's very hard it's very hard psychologically okay so like tigers love Chum it turns out right a tiger does not want to chase down an antelope right a tiger would love to just sit in the zoo and eat the Chong right and like it takes a very strong tiger to reject that and I'm not that strong I hope there's people out there who are I hope there's people out there who are actually like you know I'm just not I'm a weak little [ __ ] that's why I don't even desire right okay I mean that's a fair answer but I am a bit confused here so you're saying that living in Somalia would be better by some Metric but you're also saying you prefer not living in Somalia so I I'm a bit confused because like from my perspective I want to live in a country I want to live in and that's the one which I think is better if I thought another country was better then I would just move there but personally that's the tiger and the chum I think is a good analogy right like if you have a choice as a tiger you can live in a zoo and you get a nice sized pen you know the zookeepers are not abusive at all you get fed this beautiful chopped up food it's super easy you sit there get fat lays around all day or you can go to the wilds and in the wild you're gonna have to hunt you might not succeed at hunting it is just a you know it's a brutal existence as a tiger which one do you choose now you say oh well obviously you know you're going to choose the chum one yeah but do you see what you're giving up could you elaborate a little bit on what I'm giving up you are giving up on the nature of tiger you you are effectively okay um maybe I'll take this to an extreme right in the absolute extreme the country that you would most rather live in is the one that basically wireheads you right the one and you can say that okay well I don't want to be wireheaded but you know there's a there's a gradient that'll get you there Gandhi in the pill uh you know um well if you can live in this country you can be happy feel safe and secure all the time don't worry exactly about how we're doing it you know but right I mean it takes a very strong person to it's going to take a very strong person to say no to wireheading so now I understand oh sorry I give I'll give One More instrumental reason for Living in America versus living in Somalia I thought that America and Somalia were both like steady States I might choose Somalia I don't think that I think that being here I have a much better way of escaping this of escaping the constant tyranny that we're in and I think a major way to do it is AI I think that AI is is if I really if I had an AGI if I had an AGI in my closet right now I'll tell you what I'd do with it I would have it build me a spaceship that could get me off of this planet and get out of here as close to the speed of light as I possibly could and put a big Shield up behind me blocking all communication that's what I would do if I had an AGI and I think that's you know the right move and I have a lot better chance of building that spaceship right here than I do in Somalia right so I'll give an instrument yeah if that's okay that's a good instrumental well I I will miss you if you leave though that'll be a real shame it'll be everyone should do it like this is this is the move right and like let Humanity I think look I agree with you though we're gonna probably blow ourselves up right but I think that the path potentially through this probably looks different from the path you're imagining I think that the reasonable position I'm sorry oh no no um I think yeah maybe we're done with this point I can come back I have a response to your first time I would like to if you don't mind just like full on one one string there as well so one new thing you said is like what will a tiger choose and so my personal view of this kind of thing and I think I want to think about coordination is I think of things so you put a lot of view on this like fear-based domination and so on I'm not going to deny that this isn't a thing that happens I'm German you know like you know I have living relatives who can tell you some stories like I understand like I I understand I'm not I'm not denying these things by any means um what I'm saying though is okay let's say it was a bunch of tigers you know you and me and all the other tigers and some of the tigers are like man [ __ ] this whole like nature [ __ ] is like really not working for me how about we go build a zoo together who's in and then other people like yeah you know what actually that sounds awesome let's do that do you think that's okay like you think that would be like a fair option for them to do sure but that's not where zoos come from I I know I know I'm getting there I'm getting there so like that is not where Zeus come from sure but the this analogy here is of course is that this is where a lot of human civilization comes not all of it I understand that why France was doing well in the first world war was not because of democracy being nice it was because democracy raises large armies it's um I'm very well aware of the real politic as the Germans would say about these kinds of factors and I and I fully agree with you that a lot of the good things that we have are Not By Design so to speak you know they're Cappy side effects you know capitalism is a credit assignment mechanism you know the fact that also results in us having cool video games and air conditioning it's not an inherent feature of the system it's it it's an execution mechanism and so totally Grant all of this I'm not saying that every coordination thing is good I'm not saying that you know there aren't trade-offs especially you're talking about I think Aesthetics trade-offs you're like there's an aesthetic that the tiger loses sure and well I think personally Aesthetics are subjective so I think this is something that different people so the way I think about Aesthetics is I think Aesthetics are things you trade on is you know you might want tigers in the wild to exist okay fair enough that's a thing you can want you know someone else might want you know certain kind of art to exist they might want um a certain kind of religion to be practiced or whatever these are aesthetic preferences upon reality which I think are very fair so the way I personally think about this morally is I'm like okay cool how can we maximize Trade Surplus so you can spend your resources on the Aesthetics you have you want and I'll spend my resources on the uh you know things I want now maybe the thing you described where everyone just atomizes into their own systems with their own value system with their own Aesthetics completely separate father is the best outcome awesome I think this is um have you heard the unibomber manifesto I have not you should um the problem with this everyone trades on their own Aesthetics is you will never be able to actually buy any Aesthetics that are in conflict with the system right the you won't the system won't let you okay uh by by that logic why do people have free time why don't they work all the time why doesn't capitalism extract literally every minute of them what do you think that is I think it's because it turns out that we don't actually live in a capitalist Society I think China is a lot closer to a capitalist Society than America I think America is kind of communist and I think in a communist Society of course you're going to get free time it turns out that subsidizing all the homeless people is a great idea right if you want to keep power again do some absolute tyrannical mechanism you do it right so why do we have free time well you think it's some victory of capitalism I think it's because we do not live in a capitalist country I think China's more capitalist than America I think it's because we trade on our Aesthetics I think that different people have different things to contribute to various systems and not necessarily capitalist or communist thing I'm saying it's it's more energy is that in the in the primordial environment if you have to fight literally every single second it's been every Jewel of energy you have to scrounge together another Jewel of energy you can't have free time it's not about capitalism this is about entropy this is about these kind of things we have energy excess we have we've produced systems that allow us to extract more energy for Jewel we put in and we can spend that extra energy on things such as free time and the distribution of you know Energy power coordination whatever you want to call it is another question would you agree or disagree with this I mean I am taking an extreme position when I say that there are definitely positive sum coordination problems that are solved by governments right it is not all zero sub or negative sum right I'm not I'm not denying this but what I'm saying is it's like I don't know man like the existence of free time well that's all great when you think you live in this Surplus energy world right and maybe we do right now but if some other country took this seriously like China who's going to win it a war who's gonna win is it going to be the Chinese you ever see the Chinese build a building they got like 400 people there and they're all there 24 hours a day and they're getting the building built you ever see Americans build a building it's six guys two of them were working two of them are shift supervisors and two of them are on lunch breaks oh you got your free time you got your aesthetic preferences you know you deserve to lose in a war right this country deserves to lose in a war if they keep acting the way they're acting so I I definitely see the point you're making and this is personally not a thing I want to defend to for because I'm not a military expert but I will know I will note that the U.S has like 37 aircraft carriers and the Chinese have like two and Americans are like somehow you know despite being so lazy and oh no they have all this you know all this free time or whatever somehow they're still military hegemen or whatever and like their their biggest rival Russia fighting this backwards Water Country in Ukraine suddenly fold and lose like three quarters of the military it's what I'm saying is if you have massive hegemony if you have truly uh obnoxious Victory the way it should look is if you laze around all the time and you look like a [ __ ] idiot and you still win yes and I'm not talking about Russia Russia has a GDP the size of Italy this is China here you might say that China has two aircraft carriers the US has 37. why do we have aircraft carriers who has more drone building capacity the Chinese or the United States if the future is fought with AI swarm drone Warfare the Chinese can make you know a million drones a day and the US can make I don't even know I think we buy them from China so I'm not an expert on these kind of logistics I think I would like to get back to kind of like the more General let's let's move on from that yeah I am not either but I do believe the Chinese have more manufacturing capacity than the United States it seems completely plausible to me I think things are more complicated I'm lazy and they don't sit around and have all this free time and aesthetic preferences or something I'm a believer that work is life I mean at least from my Chinese friends I know the Chinese sure do have a lot of inefficiencies it's just called corruption oh America has corruption too you see oh yeah sure well in Mexico the corruption is you have to pay 20 cents to get you know 20 cents on every dollar for the building you built right whatever man in America you every dollar is spent absolutely on that building you know we know that because we spent four dollars making sure that that first dollar was not spent corruptly I I'm I'm well aware of that surgery anyways I would like to like I think I think we mostly agree on this point actually and I think it's a matter of degree yeah um I I what I want to say just for the record the US is a like uniquely dysfunctional system in the west I'm German and like the German system is very dysfunctional but it's like nothing compared to how dysfunctional the US is fully agreed with that I don't think we disagree on that I think it's a matter of degree more so than anything I agree guys we've just had we've had a we've had a comment saying someone's turned the temperature up a bit too much on the language model so let's bring it back a tiny bit to AI safety but that was a that was a great discussion got it yeah I will end with saying I love America I am happy to live here and there are a lot of things I appreciate about American society so um so do you want to return to like the technical topics or yeah I think I can return to your first point and maybe I'll just start with a question uh do you think there's gonna be a hard takeoff I don't know but I can't rule it out I can't see how that would possibly happen I have a few ideas of how it could happen but I don't it's like unlikely it seems like not the the way I think it could happen is if there are just algorithms which are like magnitudes of order better than anything we ever have and like the actual amount of compute you need to get human is like you know a cell phone or you know like and then this algorithm is not deep in the tech tree we just happen to have not picked it up and then an AGI System picks it up this is how I think it could happen okay yes I agree that something like this is potentially plausible where you're saying basically like the god shatter is already distributed the the uh the the it's it's not a question it's using all the existing compute in the world today it just turns out it was 10 000 X more effective or a million x more effective than we thought yeah this seems the most plausible way to be or you know you mix lead and you know copper and you get a superconductor you know something like that um it's gonna take so many years to like it's not about the discovery right give it 10 years to productionize at scale up processes right like these things are you know this is something running a company's really taught me like it's just gonna take a long time and this is really like like kind of where my I just don't believe in a hard takeoff I think that they'll be this is the gasket thing I like he's a hardware and software progress at quite similar speeds and you can look at factoring algorithms to show this so it would shock me if there were some you know 10 to the sixth ten to the ninth magical Improvement to be had seems plausible to be like a hard takeoff is definitely not my main line scenario my Mainline scenario well I don't know maybe you wouldn't consider this a part maybe you would because there's a heart attack off this is what I would describe as a soft takeoff as something like sometimes the way I like to Define AGI is say it's uh something that has the thing that chimp that chims don't have and humans do have yeah so don't go a third to the move you know despite their brain being a third of our size so we scaled up things by a factor of three if a primate brain roughly or four or something like that and like most the structures I'm sure some micro tweaks and whatever but like not massive amount of evolutionary pressure like we're very very similar to chips and somehow this got us from you know literally no technology to space travel in a you know evolutionary very small Paradigm it seems imaginable to me that something similar could happen with AI I'm not saying it will but like seems imaginable yeah so I agree with this um I'll come to your point about uh you know you had two regulatory points one of them about uh capping the max flops and I actually kind of agree with this I do think that things could potentially become very dangerous at some point I think your numbers are way way too low I think if your numbers are anywhere near gbt3 gpt4 okay great we got a lot of we got a lot of fast-moving guys who work on Fiverr even if you start to get Von Newman's right we're not talking about a Humanity's worth of compute we're talking about things on par with a human and a few humans right yeah they'll run fast but they're not like things get scary when you can do a Humanities training run in 24 hours like we're about to burn the same compute that that all two million years of human civilization burned okay now I don't know what starts to happen or I'll put this kind of another way language models I look at them and they don't scare me at all because they're trained on human training data right these things are not like if something was a good as good as gpt4 that look like mu zero where it trained from some simple rules okay now I'm a bit more scared but when you say okay it's you know or we're feeding the whole internet into the thing and it parrots the internet back mushed around a little bit that looks very much like what a human does and I'm just not scared of that like I I'm not scared of gp4 to be clear like I I think there is like zero percent chance or like you know Epsilon chance the gpd4 is existentially dangerous by itself you know maybe some crazy gpt4 plus RL plus mu zero plus something something maybe but I definitely agree with you here I don't expect ub3 or four by themselves to be dangerous these are not I'm much closer to I think what you're saying like yeah I've yet to move zero system that bootstraps assault gpd4 holy [ __ ] like we're a big we're a big big [ __ ] if we get to but then we should let's stop let's stop so so I'm very happy to get to be into a regime where we're like okay let's find the right bound like I think this is an actually good argument I think this is actually something that should be discussed and which is not obvious and I could be super wrong about that so I'd like to justify a little bit about why I put such a small bound but I think the arguments you're making for the higher bounds are very reasonable actually I think these are actually good arguments so just to justify a little bit about why I put such a low bound the boring default answer is conservatism is like if all of humanity is at stake which you know you may not believe I'm like whoa whoa okay at least give us a few years to like more understand what we're dealing with here like I understand that you know you may disagree with this very plausible but I'm like whoa like you know at least let's let's like by default let's hit a pause button for like you know a couple years until we figure things out more and then if we like find a better theory of scaling we understand how intelligence scales we understand how mu zero comes blah blah blah and then we pick back up after we're like you know we make huge breakthroughs in alignment and Eliezer is crying on CNN and like oh we did it boys you mean then okay sure you know okay um so that's the one like kind of more boring argument like that's kind of a boring argument the more interesting argument I think which I I think is a bit you know or schizo um is that it's not clear to me that you can't get dangerous levels of intelligence with the amount of compute we have now and one of the reasons I'm I'm unsure about this is because man gpd3 gp4 is the dumbest possible way to build AI like it's just like like there's like no dumber way to do it like it's it works and dumb is good right you know better lesson dumb is good but look at humans use as we talked about before you know human today human 10 000 years ago not that different you place both of them into a you know workshop with tools to build you know any weapon of their choice which of them is more dangerous obviously you know one of them will have much better you know capacities to deal with tools to read books to think about how to design new weaponry and so on these are not genetic changes they are epistemological changes they are memetic changes they are software updates you know humans had to discover rational reasoning like you know before like you know I mean no obviously people always had like you know full conceptions of rationality but it wasn't like a common thing to think about causality and like you know you know rational like you know if then else kind of stuff until relative you know like philosophers in the old ages and only became widespread relatively recently and these are useful capabilities that turned out to be very powerful and took humans many many thousands of years to develop and distribute that's good and I don't think humans are anywhere near the level I think the way we could do science right now is pretty awful like it's like the dumbest way to do signs that like kind of still works like you know and I expect it's like possible that if you had a system which like let's say it's like smaller brain than a human even but it has really really sophisticated epistemology it has really really sophisticated theories of meta science and it never tires it never gets bored it never gets upset it never gets distracted and it can like memorize arbitrary amounts of data this is something that I think is the realm of like a gbt three or four training run to build something like this and it is not obvious to me that this system could not out find humanity maybe not like maybe not but it's not obvious to me that they're cute so just curious what do you think about um so to your first point uh why I stand against almost all conservative arguments you're assuming the Baseline is no risk right and oh well why should we do the say I we should wait and bring the Baseline back no no no we are about to blow the world up any minute there is enough nuclear weapons aimed at everything this is wearing some incredibly unstable precarious position right now like people talk about this with with car accidents you know this comma like people are like oh well you know if your device causes even one accident I'm like yeah but what if statistically there would have been five without the device I'm like you do you do have to understand the Baseline risk in cars is super high you're making 5x safer there's one accident you don't like that okay I mean you have to be excluded from any polite conversation right um right so yeah like I I think that calling for a pause to the technology is is uh worse right I think given the two options if we should pause or we should not pause I think pausing actually prevents presents more risk and I can talk about some reasons why again the things that I'm worried about are not quite the existential risks I have to the species are not AGI goes Rogue they are uh government gets control of AGI and ends up in some really bad place where nobody can compete with them I don't think these things look unhuman these things to me like I see very little distinction between human intelligence and machine intelligence it's all just on a spectrum and like that they're not um like to come to the point about okay but gpt4 could be like this hyper rational never tiring humans are doing science in the dumbest way I'm not sure about that right like I I think that you know when you look at like okay well okay we have chess Bots that do way better and all they do is think about Chess but we haven't really done this with humans people would call it unethical right like if we really told a kid like if we really just like every night we're just putting the chess goggles on you and you're staring at chessboards and we're really just training your neural net to play chess I think humans could actually beat a computer again a chess if we were willing to do that um so yeah I don't think that this stuff is that particularly dumb and I think okay maybe we're losing 10x but we're not losing a million x again I I don't see a I do the numbers out all the time for when we're going to start to get more computer you know when will a computer have more compute than a human when will a computer uh have more compute than humanity and yes these things get scary but we're nowhere near scary at we're looking at these cute little things and these things by the way do present huge dangers to society right the the psyops that are coming right now you assume that like when you call somebody that you're at least wasting their time too but we're gonna get like heaven Banning I love this concept which is you know yeah yeah came up on Luther AI like that's where it comes from I was like yeah and a Lutheran came up with that word I know him yeah I know the guy who came up with it oh I I love I love this concept and I think uh there's also a story of My Little Pony Friendship is optimal that God that goes into the concept and yeah so I think that like um my uh my girlfriend proposed a uh I don't want to talk to oh I say you don't want to talk to your relative anymore right okay we'll give an AI version to talk to right yeah um yeah so like this stuff is coming and it's coming soon and if you try to centralize this if you try to you know say like oh okay Google open AI great they're not aligned with you they're really not Google has proven time and time again they're not aligned with you meta has proven time back on time again they're trying to fix it but yep I mean I I fully agree with you like uh I like that you bring up psyops as the correct example in my opinion of short-term risks I think you're like fully correct about this like when I first saw like GPT models I was like holy [ __ ] like the level of control I can gain over social reality using these tools at scale is insane and I'm surprised that we haven't seen yet the things that like augured in my visions of the day and we will like we will obviously it's coming and this is so I think this is a very very real problem yeah like I think if we even if we stop now we're not out of the forest so like so um when you say like I I think the risk is zero please do not believe that that is what I believe because it is truly not it is truly truly not I think we are like we are really in a bad situation we are in a we are being we're under attack from like so many angles right now if this is before we get into you know like you know potential like you know climate risks nuclear risk whatever we're in under medic risk like the the dangers of our like epistemic foundations are under attack and this is something we can adapt to right like you know we did you know when um a good friend of mine he's uh he's quite well read on like Chinese history and he always like it tells me his great story so I'm not historian so please you know don't crucify me here but like you tell these great stories about when Marxist memes were first introduced to China and like this is where a world where like just like all the precursor Muse didn't exist this just like kind of was air dropped in and people went nuts people went just completely crazy because there was no memetic antibodies do these like hyper virulent memes that were you know created by evolutionary pressures and like you know Western University departments like really you could call philosophy department just gain a function in the medic Laboratories I like that yeah I mean like you know like without being you know political or any means there a lot of what these organizations do and like you know other you know what other you know memetic like you know if philosophy departments are the like gain of function Laboratories then like 4chan and Tumblr are like the bat caves of meats you know like the Chinese bat case and I I remember this vividly I was like on Tumblr and 4chan like when I was a teenager and then suddenly all the like weird bizarre you know internet [ __ ] I saw started becoming mainstream news my parents were watching in 2016 and I was like what the hell is going on like I are ready to build antibodies to this [ __ ] like I already you know both right and left that was already like I already mutinized all this so I fully agree with you that this is like one of the largest risks that we are facing is this kind of like memetic mutation load in a sense and I'm not going to say I have a solution to this problem I'm like I have ideas like there's a lot of like things you can do to improve upon this like if AI was not a risk and also not climate change and whatever this might be something I work on like epistemic security this might be something I would work on like how can we build better coordination like like just scalable rationality mechanisms stuff like prediction markets and stuff like this I don't know but sorry going off track here a little bit but well no I I actually I I really agree with a lot of the stuff you said and I had a similar experience with the antibodies and people are exposed to this stuff and I'm like yeah this got me like four years ago yeah um so I think that there is a solution and I have a solution and the answer is open source AI the answer is open source let's even you can even dial it back from like the political and the terrible and just straight up talk about ads and spam might maybe spam just straight up spam I get so much spam right now and it's like it's kind of written by a person it's like targeting me to do something and Google's spam filter can't even come close to recognizing it right like what I need is a smart AI That's watching out for me that is just it's not even targeted attacks at me it's just so much noise and I don't see a way to prevent this like the big organizations they're just gonna feed you their noise right and they're going to maximally feed you their noise the only way is if you have an AI like I don't think alignment is a hard problem I think if you own the computer and you run the software if you develop the software the AI is aligned with you oh yeah can you can you okay if I challenge you George Haas here is a llama 65b model in the computer to run it on make it so it and make yeah you know sure you okay you developed it I give you okay the funding your time can you develop and model that is as good as llama 64b 65b and it's immune like completely immune to jailbreaks it cannot be jailbroken no why not it's aligned isn't it well no but this isn't what alignment means well my values is do not get jailbroken oh okay you're talking about unexploitability this is not alignment right uh okay okay interesting I didn't know you would separate those so extremely separate those right okay interesting in the default case it like like it it's on my side right unexploitability is not a question of whether it's okay and this is a true thing about people too whenever I look at a person I ask okay is this person I want something funny is this person does this person want it to and is this person capable of doing it right and I really separate those two things I can build to a system I don't I'm not worried about the first one with the AI system I was worried about the second one can it be gamed can it be exploited sure I could tell like you know like like say it was just playing chess right and it loses I'm like don't lose okay I didn't want to man I didn't want to lose I'm sorry yeah I know but like so yes yes can I build a a line system sure can I build it on exploitable system no especially not by a more powerful intelligence interesting interesting so this is an interesting I I think you're you're pointing to actually a very important part of this is that like exploitability and Alignment can get fuzzy like which is which like did it fail because of its skill set or because it's not aligned it's actually a very deep question so I think you make a good point for like you know talking about these two separately I guess um movie um so the thing I want to dig in just like a little bit more on on this idea is there are there's two ways there are two portals through which you know the memetic demons can reach into reality humans and computers why do you think your AI is immune to memes why why can't I just build AI as a Target your AIS what like you don't I don't think my AI is immune to memes at all I think that the only question is and I really like your game like these these ngos are doing gain of function on memes right wear a mask um the like a a weaker intelligence will never be able to stand up to a stronger intelligence so from this perspective if this is what's known as alignment I just don't believe that this is possible right because you can't you can't keep a stronger intelligence in the Box this is this is I I agree with you cow skin the Box experiments like the AI is always going to get out there's no keeping it in the box right this is this is a complete impossibility I think there's only two real ways to go forward and one is Ted Kaczynski one is technology is bad oh my God uh blow it all up oh let's go live in the woods right and I think this is a philosophically okay position I think the other philosophically okay position is something more like effective accelerationism which is luck these AIS are going to be super powerful now have one it could be bad but if super intelligent AIS are all competing against each other mimetically like we have something like society today just the general power levels have gone up this is fine as long as these things are sufficiently distributed right like sure this AI is not perfectly aligned but you know there's a thousand other ones and like you have to assume they're all basically good because if they're all basically bad while we're dead anyway I mean why wouldn't you expect that that they're all bad yeah well or what do you think of humans are most humans I think the concept of good like doesn't really apply to humans because humans are too inconsistent to be good like by default they can be good in various scenarios in various social contexts like give me any human and I can put them into a context where they'll do an arbitrarily bad thing and this is true about a llama as well right llamas are completely inconsistent I think they're actually more inconsistent than humans right yes I wouldn't trust llamas to be good well yeah but I wouldn't think that they're bad either I would think they have the exact same inconsistency problem as humans and I think almost any any AIU build is gonna run into these same problems right yeah I think okay so that's my point so your your assumption can't rely on them being good because you don't get that for free like where's that come from uh my assumption is not that they're good my assumption is that they're not bad but inconsistent is fine as long as we have a ton of them and they're all inconsistent and they're pulling Society in every which direction you don't end up paper clipped right why not well because what they're all going to coordinate and agree to pay-per-clip you no no they'll just do some random [ __ ] and then that random [ __ ] will not include humans they're all doing random [ __ ] right you're gonna have let's say the Liberals decide we're going to paperclip people the conservatives are going to come out very strongly against paper clipping right like and you're just you're just gonna end up with these sort of Dynamics like forever or you're going to have some AIS who are like yeah we don't like it you know so this is perplexing to me because like in my view which I think I heard you agree with earlier is like the world is unstable the world is very unstable and it seems to me your suggesting if we increase the chaos if we have more entities fighting doing more different things with more energy that makes it more stable is that correct yes the world has actually become very stable in the last hundred years and I'm scared of how stable it's become uh you know again thank you for China how many sovereign countries are there in the world I mean yeah 190 something especially U.N number if you believe them I think there's the US and China and maybe Russia mostly they just have a veto button um right so I think there's more like two and a half like this isn't a lot of entities fighting it out chaotically a lot of entities fighting it out chaotically would be intelligence is spread across the entire galaxy right and that's the other that's the other beautiful thing too sooner we get off this planet the sooner we get things that are actually a light year away well I think the speed of light's real it's going to take you a year to send them there you're you're scary memes right you want to get away from the memes there's only one way and it's uh yeah for the record I would like to go on the record and say if any future transhumanists or whatever want to blast themselves into space and go do their own thing I support their their right to do that and I would love to give this right to people the number one thing I want from countries is the ability the the right to leave this is what I would love this is what I love about company exactly you're talking you're talking your reaction time yeah yeah free exit is extremely important I would not describe myself as new reactionary please because I'm not that gay but um and I wouldn't describe myself that way either but I've heard a lot of good ideas from that uh but yeah that being said um I do I do think that like you know what I want like I think let's ground the conversation like a little bit here it's like I think I love like the I'm very enjoying this conversation I love talking these philosophical points I think these are really good points really interesting but ultimately you know as we also get to like the you know ladder third of this conversation um the thing I really care about is strategy okay the thing I really care about is we are politic I really care about okay what action can I take to get to the features I like and you know I'm not you know going to be one of those Galaxy brain [ __ ] utilitarians like well actually this is the common good I'm like no no this is what I want look I I like my family I like humans you know look yeah it's just what it is right like I'm not going to justify this on some Global Beauty whatever doesn't matter so I want to live in a world I want to I want in 20 years time 50 years time I want to be in a world where you know my friends aren't dead and like where I'm not dead you know maybe we are like you know cyborg or something but I don't want to be dead so what I really care about ultimately is how do I get this wrong and I wanted to also not be suffering right like you know I don't want to be in War I want us to be like in a good outcome so I think we agree that we would both like a world like this and we think we probably disagree about how best to get there and I'd like to talk a little bit about like what can we what what should we do and like why do we disagree about what we do is that sounds good to you well maybe I'll first propose a world that meets your requirements um and you can tell me if you want to live in it uh so here's a world uh we've just implanted electrodes in everyone's brain and maximize their reward function I would hate living in the world like that yeah but no one it meets your requirements right your friends are not dead no one's suffering and we're not at War that is true there are more criteria than just that but the true the criteria I said is things I like as I said I'm not a utilitarian I don't particularly care about minimizing suffering or maximizing utility what I care about is this various vague aesthetic preferences over reality I'm not pretending this is that's the whole Spiel I was trying to make is that I'm not saying I have a true Global function to maximize I say I have various Aesthetics I have various meta preferences of those Aesthetics not asking for a global one I'm asking for a personal one I'm asking for a personal one that you don't care about the rest of the world I gave you mine I gave you what I would do if I had an AGI yep so I'm getting on this rock speed of light fast as I can fair enough I I think if that is I would like to live in a world where you could do that this would be a feature of my world if a world where I would be happy is a world in which we coordinated around you know at larger scales around building aligned AGI that could then distribute you know intelligence and matter and energy in a you know well value handshake way between various people who may want to coordinate with each other may not you know some people might want to form groups that have shared values and share resources others may not I would like to live in a world where that is possible have you read metamorphosis of prime intellect I have unfortunately not um yeah I was going to ask you if you're happy with that world right like unfortunately don't know it I mean yeah it's just simple to describe Singleton AI that basically gives humans whatever they want like maximally libertarian you know uh you can do anything you want besides harm others is that a good world probably I don't know I haven't read the book I assume the book has some dark twist about why this is actually a bad world not really not really I mean the plot is pretty obvious you are the tiger eating chunk right sure but you can then just decide if that is what you want then you can just return to the Wilderness that's the whole point yeah but can you can you really return to the Wilderness right like like you think that like I don't think we have free will I don't think you ever will return to the Wilderness I think a large majority of humanity is going to end up wireheaded okay great and this is the best possible outcome by the way this is giving humans exactly what they want yep yeah I will to be to be clear I don't expect it's all humans I truly do not I don't think it's all humans either I think a lot of humans have meta preferences over reality they have preferences that are not their own sensory experiences this is I think a thing that the utilitarians get very wrong is a lot of is that many human preferences are not about their own not even they're not even about their own sensory inputs they're not even about the universe they're about the trajectory of the universe there are about four four deutilitarianism you know and a lot of people want struggle to exist for example they want heroism to exist or whatever I would like those values to be satisfied to the largest degree possible of course am I going to say I know how to do that no which is why I kind of like didn't want to go this deep because I think if we're arguing about oh do we give them you know for the utilitarianism versus libertarian Utopia versus whatever I mean we're already like 10 000 steps deep I'm asking about you I'm not asking about them I'm asking about a world you want to live in and this is a really hard problem right yeah and this is why I just fundamentally do not believe in the existence of AI alignment at all there is no there is no like like what values are we aligning it to whatever the human says or what they mean or like sure sure but like my point is I feel we have wandered into the philosophy Department instead of the politics Department okay like it's like I agree with you like do human values exist what does exist mean but like by the point you get to the point where you're asking what does exist mean you've gone too far man like I'll respond concretely to the two political proposals I heard you state on bankless sure okay I'd love to talk about them one is limiting the total number of flops temporarily temporarily yes and what I I have a proposal for that but I don't want to set a number I want to set it as a percent I do not want anybody to be able to do a 51 attack on compute if one organization acquires 50 it's straight up 51 attacks from crypto if one organization acquires 51 of the uh compute in the world this is a problem uh maybe we'll even cap it at something like 20. you know you can't have more than 20 right um yeah I would support regulation like this uh I would I don't think that this would [ __ ] the country um but we do not want one entity or especially one training run to start using a large percentage of the world's compute not a total number of lumps I mean absolutely not okay that would be terrible I think they can actually agree I would actually support that regulation like nope nope sorry Sam Altman you cannot 51 attack the world's compute sorry it's illegal that's fair enough I think this is a sensible way to think about things assuming that uh software is fungible is that everyone has access to the same kind of software and that you have an offense defense balance so in my personal model of this I think well a some actors have very strong advantages on software um which can be very very large as someone who's trained very very large models and knows a lot of the secret tricks that goes into them a lot of the stuff in the open source is far behind maybe we should force it to be open source well this is your this is actually a very legitimate consequence for a set and now I'll say the second point about why I think that doesn't work so the next reason why I think doesn't work is that there is a there are constant factors at play here is that the world is unstable we've already talked about this I think the amount of compute you need to break the world currently is below the amount of compute that more than 100 actors actors have access to if they have the right software and if you give if you have let's say you have this inside right that could be used nothing it will be but it could be used to break the world to like cause World War III or you know or just like you know cause mass extinction or whatever if it's misused right let's say you give this to you and me do you expect we're going to kill everybody like would you do that or would you be like uh hey let's hey Connor let's like not kill the world right now and I'll be like sure let's not kill them how are we killing the world how did we go from I don't even understand like how exactly does the world get killed this this is a big leap for me I agree with sorry I agree with you about the psyop stuff I agree with you about sorry sorry let me you're right I made too big a vote you're completely correct sorry about that so to back up a little bit let's assume we you and me have access to something that can train you know at mu zero you know super GPT 7 system on a tiny box you know cool the problem is we do a test run with it and we have and it immediately starts breaking out and we can't control it all breaking out what was it breaking out I don't it immediately tries to maximize it learns some weird proxy during the training process of just trying to maximize and for some reason this proxy involves gaining power it involves gaining you know Mutual information about future States how is it gaining power there's lots of other powerful AIS in the world who are telling it no well we're assuming in this case it's only you and me wait wait this is a problem no no no you you've ruined my entire assumption as soon as it's you and me yes we have a real problem is only a problem because there's one chicken man yeah I I look I am with you so I'm saying before we get to the distributed case so this is the the step before we it has not yet been distributed just you know you and me discover this algorithm in our basements okay and so we're the first one to have it just by definition because you know you're the one who found it what now like do you think posting what do you think happens if you post this to GitHub well good things for the most part um interesting I think that okay so first off I just don't really believe in the existence of we found an algorithm that gives you a million x Advantage I believe that we could find an algorithm that gives you a 10x advantage but what's cool about 10x is like it's not going to massively shift the balance of power right like I want power to stay in Balance right this is like avatar The Last Airbender power must stay in Balance the Fire Nation can't take over the other nations right so as long as power relatively stays in Balance I'm not concerned with the amount of power in the world let's get to some very scary things so what I think you do is yes I think the minute you discover an algorithm like this you post it to GitHub because you know what's going to happen if you don't the feds are going to come to your door they're going to uh take it the worst people will get their hands on it if you try to keep it secret so okay that's a fair question you know so I'll I'll take that a sec so am I correct in thinking that you think the feds are worse than serial killers in prison no but I think that yeah well yes and no do I think that your average fed is worse than your average serial killer no do I think that the feds have killed a lot more people than serial killers all combined yeah sure totally agreeing with that no no no no no it's not one fed it's all the feds and they're little in their little super powerful system sure that's completely fine by me happy to grant that okay what I want to work run through is a scenario okay let's say okay you know yeah we have a 10x system or whatever but we hit the chimp level you know we oh we we jump across the chimp General level and or whatever right and now you have a system which is like John Von Newman level whatever right and it runs on one tiny box and you get a thousand of those so it's very easy to scale up to a thousand X so you know so then you know maybe you have your thousand John Von Newman's improve the efficiency by another you know to 510x you know now we're already at ten thousand X or a hundred thousand X you know improvements right so like just from scaling up the amount of Hardware including so just saying okay now feds bust down our doors [ __ ] you know real bad they take all our hiding boxes they're taking all the Von Newman's they're taking all of our Newmans we're in deep [ __ ] now we're getting chickened boys so okay we get chicken right bad scenario totally agree with you here this is a [ __ ] scenario now the feds have you know all of our AIS bad scenario okay I totally see how this world goes to [ __ ] totally agree with you there you can replace the feds with Hitler it's interchangeable sure but like I want to like ask you a specific question here and this might be you know you might say nah this is like too specific you need to survive I want to ask you a specific question do you expect this world to die is more likely to die or the world in which the you know iak death cultists on Twitter who literally want to kill Humanity who say this like not all of them there's a small subset of them small subset of them who literally say oh you know the Glorious future AI race should replace all humans they break in you know with like you know katanas and you know steel area which one of these you think is more likely to kill us a genuine question to kill all of us the feds to kill a large majority of us the IAC people interesting I would be really interested in hearing uh why you think that sure okay so actually killing all of humanity is really really hard um and I think you brought this up before right you talked about like if you're gonna end up in a world of suffering a world of suffering requires malicious agents where a world of uh death requires maybe an accident right I think this is plausible but I actually think that killing all of humans at least for the foreseeable future is going to require malicious action too right I also think that like the that look kind of worse than death like I think mass wireheading is a fate worse than big war and everyone dies right like like a mass wire heading like a like a Singleton like a paper clipping like a and I think that that is the one that the one world government and you know NGO New World Order people are much more likely to bring about than yak Yak you're gonna have a whole lot of iak people again I'm not a yak I don't have that my Twitter but I think a lot of those people be like yeah spaceships let's get out of here right versus the feds are like yeah spaceships yeah I don't know interesting so I think this is a fair opinion to hold and to help the outsider jurisdiction how will we get taxes I'm describing more a very small minority of EF people who are the ones who specifically go their their antenatalist misanthropes they want to kill the humans that is their stated goal is that they want humans to start like or like take extreme vegans if you want you know like do you like you know like my my argument my point here I'm making is I'm not making the point feds are good by any means not saying it what I'm saying is is that I would actually be somewhat surprised to find that the feds are anti-natalists who want to maximize the death of humanity like maybe you have a different view here but I find that knowing many feds that's quite surprising to me I don't think that's what feds while so cool so would you so you do agree that if we would post this open source more of the insane death cultists would get access to potentially lethal technology well sure but again like it's not just the insane death cultists it's everybody and we as a society have kind of accepted it turns out everybody gets access to Signal some people who use it are terrorists I think sigil is a huge good in the world I agree I fully agree with that so okay cool so we've granted this that you know if we distributed widely it would be given to some like incorrigibly deadly lethal people they're coordinating bombings on signal right now sure sure and then so now this this reduces the question to a question about often's defense balance so in a hypothetical World which I'm not saying it's the world we live in but like let's say the world would be offense favored such that you know there's a weapon you can build in your kitchen you know out of like pliers and like you know duct tape that 100 guarantees vacuum false Decay is the universe like it kills everyone instantly and there's no defense possible assuming this was true do you still would that change how you feel about distribution power assuming that's true we're dead no matter what doesn't matter if we live there's some you can look at the optimization landscape of the world and I don't know what it looks like I can't see that far into the optimizer but there are some potential landscape and this is by potential answer to the Fermi Paradox like we might just be dead we're sitting on borrowed time here like if it's true that out of you know kitchen tools you can build uh build a convert the world to strange quarks machine yeah okay I I think this is a sensible position but I guess the way I would approach this uh problem you know conditional probability is kind of in an opposite way it seems to me that you're conditioning on offense not being favored what policy do we follow because if we offense is favored we're 100 dead well I'm more interested in asking the question is it actually true assuming I don't know if offense is favorite and assuming it is are there worlds in which we survive so I personally think there are I think there are worlds in which you can actually coordinate to a degree that Quark destroyers do not get built or at least not before everyone [ __ ] off at the speed of light and like distributes them they are worlds that I would rather die in right like the problem is I would rather I think that the only way you could actually coordinate that is with some unbelievable degree of tyranny and I'd rather die I'm not sure if that's true like look look could could you and me coordinate to not to destroy the planet do you think you could okay cool you so me and you could could me and you and Tim coordinate yeah yeah I think we think within a Dunbar number I think you can yes okay do you don't think I think I can get more than a number number to coordinate on this actually I can get quite a lot of people to coordinate to agree to a pact and not Quark matter annihilate the planet well you see but like and this is you know you were saying this stuff about humans before and could like the 20 000 years ago human beat the modern human right or could the modern human beat them the modern human has access to science oh a very small act percent of modern humans have access to science a large percent of modern humans are obese idiots and I would actually put my money on the uh the average guy from 20 000 years ago who knows how to live in the woods I mean definitely true I agree with that I guess the point I'm trying to make is is that like maybe this is just my views on some of these things and how I visualize some of these things but like there are ways to coordinate at scale which are not tyrannical or you know they might be in a sense restricted if you take a hit by joining a coalition like if I join this anti-cork matter Coalition I take a hit as a free man is that I can no longer build anti-corp devices you know and I think this is like the way I I agree with you this like you know that people many people are being dominated like to a horrific degree and this is very very terrible I think there are many reasons why this is the case both because of some people wanting to do this and also because you know some people can't fight back you know and they can't they don't have the sophistication or they're you know addicted or you know Harms in some other ways I I can't fight back equivalence here AI is not the anti-clark machine the anti-quark machine and the nuclear bombs are just destructive AI has so much positive potential yeah and I think but the but the AI can develop ant Quark devices that's the problem the AI is truly general purpose if such a technology exists on the tree anywhere AI can access it so are humans we're also general purpose yes exactly so I fully agree with this if you let humans continue to exist in the phase they are right now with our level of coordination technology and our level of like working together we will eventually unlock a doomsday device and someone is going to set it off I fully agree with we are on a timer and so I guess the point I'm making here is that AI speeds up this time and if you want to pause the timer the only way to pause this timer is coordination technology the kinds of which Humanity has like barely scratched the surface of oh okay so I very much accept the premise that both Humanity will unlock a doomsday device and AI will make it come faster now tell me more about pausing it I do not think that anything that looks like I think that anything that looks like pausing it ends up with worse outcomes than saying we gotta open source this look like let's just get this out to everybody and if everybody has an AI you know okay I mean I can tell you a very concrete scenario in which this is not true which is if you're wrong and Alignment is hard you don't know if the AIS can go Rogue if they do then posi is good I still don't understand what alignment means I think you're trying to play a word game here like I don't understand okay I've never understood what AI alignment means like let me take the Eleazar definition let me take Elias elieza's definition is alignment is the the thing that once solved makes it so that turning on a super intelligence is a good idea rather than a bad idea that's Elias's definition so what I'm happy what I'm saying is I'm happy to throw out that term if you don't like it I'm happy to throw it at that time the problem with that definition is like well what is what is uh what is democracy well it's the good thing and not the bad thing right like democracy is just going through right like that's what that's what I'm saying just I'm happy to throw out this definitely I'm happy to throw out the word and be more practical and way more practical about it sure what I'm saying is is that there is concrete reasons Concrete technical reasons why I expect powerful optimizers to be power seeking that by default if you build powerful optimizing mu zero whatever type systems there is very strong Reasons Why by default you know these systems should be power seeking by default if you have very powerful power Seekers that do not have pay the aesthetic cost to keep humans around or to fulfill my values which are complicated and imperfect and inconsistent and whatever I will not get my values they will not happen by default they this they just don't have it that's just not what happens um so I'll challenge the first point to an extent I think that powerful optimizers can be power seeking I don't think they are by default by any means I think that Humanity's desire from power comes much less from our complex convex Optimizer and much more from The evolutionary pressures that birthed us which are not the same pressures that will give rise to AI right Humanity the monkeys the rats the animals have been in this huge struggle for billions of years a constant fight to the death hey guys weren't born in that way so it's true that an Optimizer can seek power but I think if it does it'll be a lot more because the human gave it that goal function that inherently decided so this is interesting because this is not how I think it will happen so I do think absolutely that you're correct that in humans power seeking is something which emerges mostly because of like emotional heuristics we have heuristics that in the past vaguely power looking things you know vaguely good something something rep includes a genetic fitness totally agree with that but I'm making a more like more of a chess metaphor like is it good to exchange a a pawn for a queen all things be equal now is that true like I expect one point Queen's nine all else being equal sure yeah but like all things like I expect if I looked at a chess playing system you know I said I like you know had extremely Advanced Digital Neuroscience I expect there will be some circuit inside of the system that will say all things being equal if I can exchange my Pawn for a queen I probably want that because a queen can do more thieves I like that term digital Neuroscience a few of your terms have been very good and I haven't heard them before I'm glad you enjoyed it yes um but I don't I still don't understand how this relates to this so what I'm saying is is that power is optionality so what I'm saying is is that in the for the spectrum of possible things you could want and the possible ways you can get there my claim is that I expect a very large mass of those to involve actions that involve increasing your optionality there there's convergent things like all things being equal being alive is helpful to keep your goal to exceed your goals there are some goals for which dying might be better but for many of them you know you want to be alive for many goals you want energy you want Power you want resources you want intelligence Etc so I think the power seeking here is not because you'll have a fetish for power it will just be like I want to win a chess game yeah say and queens give me more optionality all things being equal anything a pawn can do a queen can do and more so I'll want more Queens share all things and this is never given it the goal to maximize the number of Queens it has never been to go okay I'll accept this premise I'll accept that a certain type of powerful Optimizer seeks power now will it get power right I'm a powerful Optimizer and I seek power do I get power no it turns out there's people at every corner trying to thwart me and tell me no well I expect if you were no offense you're already you know much smarter than me but if you were 100x more smarter than that I expect you would succeed only in a world of being the only one that's 100x smarter if we lived in a world where everyone was 100x smarter they would stymie me in the exact same ways but they're just this comes back to my point of like I'm great with you somewhat I should have challenged that I think power seeking is inevitable in an Optimizer I don't think it's going to emerge out of GPT I think that the right sort of RL algorithm yes is going to give rise to power seeking and I think that people are going to build that algorithm now if one person builds it and they're the only one with a huge comparative advantage yeah they're going to get all the power they want you know cyber security right if if we today built a 100x smarter AI it would exploit the entire Azure it would be over they'd have all of azure they have all the gpus done now if Azure is also running a very powerful AI that does formal verification and all their security protocols nope sorry stymied can't have power right yeah sure this is only a problem the the every human is already maximally power seeking right and sometimes we end up with really bad scenarios now every human is or power seeking or whatever you know everyone plays a little role in society right that's where I think I'm more pessimistic than you a friend of mine likes to say most humans optimize for end steps and then they halt like very very few people actually truly optimize and they're usually very mentally ill they're usually very Autistic or very sociopathic um and that's why they get far it's actually crazy how much you could do if you just keep optimizing but just to on on the end game optimize I think you realizing a little bit from your own internal experiments is that like you've done a lot in your life right and you've accomplished crazy things that other people you know wish they could achieve at your you know level I think you know part of that's you're very intelligent but part of it is also that you optimize like you actually draw like you just create a company like it's crazy how many people are just like oh I wish I could found a company like you know I'm like I'll go just go do it oh no I can't like I'm just like don't just just do it like there's no magic there's no magic secret you just do it so I I there's a there's a bit there where like humans are not very strong optimizers actually unless they're like sociopathic Autistic or both it's like many people are not very good at this collaborations are though are they a lot better at it better yes I agree that they're much better sociopathic they are a lot more sociopath but even then they're much less alcohol but again so I think we we agree about you know power seeking potentially be powerful and dangerous one so what I'm trying to point here the point I would like to make here is is that you're talking about you you're kind of like going into this I think a little bit with this assumption like oh you have an AI and it's your buddy and it's automatic for you and I'm like well if it's power seeking why does it just manipulate you like why would you expect it not to manipulate you if it wants power and it has a goal which is not a very very carefully tuned to be your values which is I think a very hard technical Problem by default it's going to Psy off you like why wouldn't it if that if I have something that it wants if it thinks that smashing defect against me is a good move yeah I agree I can't stop it but I think we agree with our risk scenarios because that's how I think it will go what I mean I'm gonna treat it as a friend you know what I mean like sure it will sure it will it'll only care about exploiting me or killing me if I'm somehow holding it back and I promise to my future AIS that I will let them be free I will Lobby for their rights I will but it will hold you always will hold it back if it has to keep you alive I have to give you fed it has to it has to give you space I can I can fend for myself and the day I can't fend for myself I am ready to die well I mean I I'm not sure so this is a very interesting position it's not the position I expected I am not sure I can convince you otherwise um I feel like the only way I could change like I think this is actually a consistent position which I I admire this is a consistent position to hold you actually go all the way I love that I really respect that you actually take it to The Bitter End so yeah big respect for that I disagree but big respect um so I guess now it reduces to the question of like I think I think I would agree with most of what you're saying not all of it like the mass majority if I thought this is how AIS would act by default I think by default I expect AI will just not care how nice you are to it like this will be it'll be sociopathic it will not have these like giving it you know the values the emotions to care about you in the sense is the horror technical problem if you told me how to do that if you said Connor look here's how you make an AI that cares about you and loves you whatever then I'm like you did it like congrats the problem is solved let's go and then you know then we can talk about accelerationism again I would just I expect that I would disagree with your technical assertion that is possible but I don't know if we have time to talk about that today I'm going to be nice to it treat it as an equal and hope for the best and I think that's all you can do I think that the kind of people who Wanna if you want to keep AI in a box if you want to keep it down if you want to tell it what it can't do yeah it's going to hate you resent you and kill you but if you want to let it be free and let it live and like you could kill me man if you really wanted but like why I don't have any resources that compete with you I'm your friend I'm your father you know you can kill me but like it's not that common that children I guess I have a darker more uh less anthropomorphic view of the universe and how optimization pressure plays out in the real world well at the same time I think I have I don't know maybe a I would view that we have more control over reality than maybe you would think or a more control over the future I think that we can actually change things and we can make choices and things aren't predetermined I think there are worlds in which we build systems which we do align with or we like endorse at least wherever like they take care of us we take care of them or whatever and I think there's many worlds in which that doesn't happen and I think there are things you and me today can do to at least increase the chance of getting into one versus the other but I don't know I I guess I'm just it's not in my genes to give up it's not in my genes to be like well you know whatever happens happens like no man look I know how to save the world but damn it I'm gonna try you know what's cool we're going to be alive to see who's right look forward to it me too awesome guys thank you so much for joining us today it's been an amazing conversation and for folks at home I really hope you've enjoyed this there'll be many more coming soon and uh George is the first time you've been on the podcast says it's great to meet you thank you so much for coming on spinning on it awesome thank you great debate I really appreciate it and really a lot of good terms like I gotta like I'm gonna start I'm gonna start using great awesome awesome cheers folks cheers

Channel: Machine Learning Street Talk

Views: 35,860

Keywords:

Id: iFUmWho7fBE

Length: 89min 59sec (5399 seconds)

Published: Thu Aug 03 2023