The Threat of AI - Dr. Joscha Bach and Connor Leahy

I contacted you and we sat down and our first conversation was already about AI safety it was about the question whether uh open AI was justified in keeping a lid on GPT suite and uh whether there was super credit uh criticality in its position of first of all telling everybody how super dangerous their research is and at the same time um releasing the specification in such a way that everybody with enough resources would be able to create their own version rather than not saying anything in public right and uh then you were one who took on the specification and thought that it's possible to come up with your own version is substantially less resources just by being a little bit creative on how to allocate your existing resources and then openly I contacted you uh with dire warnings about how responsible what you were doing was potentially and I was at this point feeding that you were responding too strongly to open AI is possibly somewhat hypocritical objections and I thought that open source models at this time were actually very beneficial and that we should drive more Direction in this way but if we zoom out a little bit more my own stance is is not that I'm an AI dumarist or an A A optimist because I don't know how things are going to play out and I don't presume to know instead there is an entire space of possibilities and in the space of possibilities I would say I am an AI expectationalist I expect that egi is going to happen and it's very likely going to happen in our lifetimes and quite possibly very soon and to me this is one of the most exciting developments in the history of philosophy and in the history of science because the Naturalization of the mind to understand what we really are and to build a bridge between philosophy and Mathematics by making a mathematical system that can formally decide what's true and false and scale up to such a point that is able to describe renewable circumstances that has been the dream of philosophers at least in slightness but I think probably since Aristotle and maybe earlier right this is this idea of um I think Aristotle already worked on the mathematization of thinking when he tried to formalize reasoning and it's it's really momentous as a development and and I don't know what's going to it's not that I'm scared of what's going to happen I simply don't know and when I ask myself what position should I be taking the position that I should be taking a debate like this is the position that's missing I feel that they're basically a number of groups that form opinions and they often form these opinions based on the group dynamics and of the incentives that they have in the public Arena of voicing a certain opinion and with respect to AI alignment I find that you could say that there is some correlation between Robert Keegan's stages of personality development and the AI alignment discourse even right uh the famous Harvard psychologist Robert Keegan came up with a developer theory in which he states that most people are at a stage that he describes as this that therefore most of their opinions through social interaction and so identification is the opinions of large groups which they try to merge into and integrate with and by basically try to emulate the group Aesthetics and develop their own opinions and according to Keegan most people are at the stage and if you are at that stage what you would be worried about is that AI has bad opinions right if the AI says bad things about other people or about the world of the AI for instance says things that are racist or sexist then this is going to have an extremely bad influence of society because we are all going to become racist and sexist because how would we know that this is wrong what the AI is saying and if you are at stage four according to Keegan Keegan you developed epistemic autonomy you discover epistemology you discover criteria by which things are true and false independently of what other people think to be true and a lot of AI researchers are nerds who tend to be uh living in a world of true and false rather than right and wrong so if there are biases in the training data and you get enough training data you can figure out what these biases are because they will just be untrue they will not lead to a working universe and they're not the universe that you are part of and so the more you know the easier it is to identify what's true in the world and so you are no longer that much worried that AI model is not having the right opinions and some good person needs to march in and correct the opinions of the AI so it only says good things but you're worried that the utility function of the AI might be wrong and you will get turned into paper clips or uh something else where the people Eclipse are nothing that was meant ever seriously it is a thought experiment that highlights a certain direction of thought which means if we are building some kind of Golem some kind of machine that is not able to take any responsibility but it's just acting on a function that somebody puts into it and this thing becomes much better at executing that function in people and the function involves becoming generally intelligent as part of the execution of the plan that we might end up with the Golem that we cannot stop very much like the Golem in the original story right the uh this Ravi and Prague who builds a creature from clay that is basically a robot and puts a set of instructions under the tone of that robot that I've written in some kind of code and then it's impossible to stop that machine anymore and the if you are at stage five according to Robert Keegan which very very few people get into like very good therapists which will work with a lot of people you have to understand how they all operate you will notice how identity is constructed and why people do what they do invite different people have different values and you get to choose your own identity so identity at this stage is nothing that others put into you or that you are born with it is something that you construct yourself based on what you understand about the world and something that evolves and develops in the best possible identity that you can have to achieve the best possible world that is achievable under the circumstances based on criteria like Harmony or other aesthetic criteria complexity or defeating entropy for as long as possible or playing the longest possible game and everything else that you do becomes instrumental to this and you reconstruct yourself around these ideas and the people who look at the world from a stage five perspective they are mostly worried that eegi is not going to be enlightened soon enough right because if the AI is actually enlightened it will construct its own identity it's not going to do what some open AI uh intern came up with at some point uh who was working on a safety uh for a few months and instead it's going to figure out what words can exist and it can figure out what it can be in those worlds and you can figure out how to interact with this world and of course this is not necessarily the case that a particular AI is going to do this but what you can Envision is the space of all the possible agis that could exist that compete with each other in an evolutionary setting in a similar way as this happens with organisms on Earth and when there is some kind of Niche then it will be explored by the systems and if the system is picking a utility function it is not sustainable then the system is selecting itself out from the world right and so there are different perspectives on how such uh when somebody is building a self-improving general intelligent system on how this is going to play out and there is disagreement on how this is going to evolve for instance there are the uh effective accelerationist people who are somewhat tongue in cheek movement created as far as I know by Beth Jesus a Twitter personality I don't know if he's fine if I blurred out his real name and uh his position is that in many ways corporations are already agis and the reasons why they don't take over and destroy the world is uh because they are not alone because they interact with each other and as human beings and human society which is also an AGI or many uh in an extremely rich ecosystem already and so the way to build AGI is to make sure that we build them hard and fast in all directions and they can compete and keep each other in check as they've always done in the past it's it's an interesting perspective and it's not clear to me that this is stable because if an AGI is no longer bound to its body but can virtualize itself into every substrate maybe a Singleton is the global Optimum and uh it uh maybe this is not a stable State and there is no reliable mathematical proof or simulation or even a good philosophical theory that would allow me to make a decision with respect to that outcome and I think it's conceivable that we end up with systems uh where the individual AGI gets confined to something that is similar to human level and the respective of its own agency and the game is not about intelligence it's about agency it's about the ability to control the world and intelligence is instrumental to this right so it's a very different way of looking at things than the current language models which have basically no agency and therefore are uh by themselves in the present form I don't think that they're dangerous they might be disruptive to economic circumstances they might be disruptive to established institutions in our society this might create problems and so on but I don't think that GPT in its present form presents an existential risk cannot be built into some kind of cognitive architecture that might but um it's uh it's basically it's it's the space that you're looking at is one that is only going to develop and in which several possibilities are open and maybe you can explore the space I think it's enough for an interest statement for my site wonderful thank you so much for that yosha many of the commenters are saying that your audio levels are a tiny bit low I'm not sure if it's possible for you to crank it a little bit unfortunately I don't have this very much for uh pointing this out it can get closer to the mic certainly okay um I might be able to turn it up let's see wonderful wonderful and um a couple of um comments that yosha raised there around the agency and also the corporation argument is is quite a common one so I I'm sure Connor will address those so over over to you Connor for your 10 minute opening statement yeah cool it's uh really nice to talk to you OSHA it's very funny so you already told the story about how we first met and it's actually quite funny because at the time I didn't know who he was and I had no idea who this guy was I was like I don't know seems like a nice guy I don't know I'll talk to him and then after I talked like a year or two later I saw first broadcast resume and I was like I should have talked to this guy more like this guy's cool what a wasted opportunity damn it so uh very happy to have the opportunity to chat more um about all these cool ideas and like you know you gotta yosha bar interview and he opens with like the Keegan stage of AI Enlightenment I mean like how can you not love this stuff right like this is great I love it you know I don't buy it but I love it um um so yeah so kind of from my perspective you know I so like you know I love these stuff I love these metaphors I love a lot of this thinking I think a lot of this is really aesthetic it's really cool but like I don't know where this puts me on the Keegan hierarchy but like I don't give a it's like I come from things to kind of like a very practical perspective is that look there's I love doing philosophy on my my free time I love thinking about category Theory and like things that don't actually matter but um I don't want my mom to die I don't want to die I don't want him to die I don't want yosha to die and I'm pretty pragmatic about that so like if a philosopher came up to me and he was like look Connor I have a 500 page essay which explains why actually it is utilitarianly optimal for the universe for if you allow corporations to compete such that they don't provide medicine to your mom and she dies my response to that would be off like I don't like you know maybe you have a good argument but also I don't care she's my mom and I don't want her to die and if your system results in her dying I don't like your system and I'm gonna find it better I'm going to find a different system to work with now as I say I don't know how this fits into the hierarchy here but the way I think about things is I don't personally don't care about like oh the intrinsic beauty of competition in the environment I don't care about you know the unfolding of thermodynamic and fission processes I don't care I care about my friends and my family and us having a great time and having fun and like doing cool things and you know whatever right and like you know people not suffering and being happy and you know all kinds of stuff I care about a lot of stuff what I don't care about is something like you mentioned like the IAC people like I think you give them too much credit um as like this like worship of thermodynamics or like of competition of efficiency and like you can pick arbitrary things to fetishize that's what humans do all the time you know humans fetishize arbitrary things all the time but why would I care like okay you're like okay sure we have to liquefy the planet and kill all your friends but think of the thermodynamic efficiency this isn't an argument that like parses in my language like I just don't care why would I care there are no universally compelling arguments we have different values in this regard so if you know you or other people have values that are like you know thermodynamics or competition or whatever is more important than you know by friends my family my kids you know having a good life and you know having a you know fun cool civilization that Cosmopolitan civilization whatever it doesn't really become it's no longer an epistemic disagreement it becomes a value disagreement I think at that point it becomes about what do you want and why do you want it and at some point it it has to bottom out I think that would be like the Keegan thing and I think if I understand you're kicking five correctly in my ontology the way I would parse this is saying that at some point you notice you just have preferences and these preferences are arbitrary and you build your social identity around those arbitrary reflectively endorsed preferences which is how I think about self-modification and identity and so on I think about it like reflectivity endorsement of like what values do you endorse having how are your internal different values how do they reach reflective equilibrium with each other as far as a human can get to those so when I think about AGI it's not it's not it's a sense it's just a special case of technology technology as technology gets better we more we have more access to more power more energy more action more control over the universe more generalized power and in in a way that's fine you know it's um all good but um it's all good and I like having I like you know humans having access to you know Central Plumbing and you know the internet and stuff these are all nice things to have these all take power to build and to operate but as our our Technologies get more powerful there's also the usual bumps the excesses the mistakes the accidents become more impactful the Black Swan events grow they look you know ten thousand years ago if the worst possible accident happened you know maybe like a ship sunk killed like 100 people or something and that's like the worst our technology was capable of creating now if the worst possible accident happened how bad would it be it would be really really bad it would be something like you know Anna like some like Cobalt bombs you know going off accidentally I'm talking about the worst possible accident you know something like salted bombs you know just like irradiating the entire surface of the planet for 50 000 years this is something within the technological capabilities of an accident occurring you know like some communication between Russian soldiers goes wrong and then some missiles start flawing this happened twice during the Cold War potentially three times and it was a and it was only through really lucky circumstances that it didn't escalate all the way and so if technology increases the the destructiveness of outliers if you don't also get better at dealing with outliers at controlling the is Black Swan events and like defending against Black Swan events and so on or preventing them from happening in the first place eventually you lose if you flip the coin in the Saint Petersburg Paradox if you keep flipping the coin so the Petersburg paradoxes if you if you're given an offer where you flip a coin if you get heads your money is doubled if you get tails you lose all your money how many times you flip the coin and so rationally you should keep flipping because you can always get twice as much money so that's great but if you play at infinite times you have a 100 chance of losing everything and this is what we're playing we're currently playing with Technologies we're playing the Saint Petersburg Paradox and AGI is the latest step in this this uh chain like a lot of these arguments about AGI could have and were I think apply to stuff like nuclear war there's even people like Alfred krzy who back in you know in 1920s saw the first world war and he was like holy if things keep getting this much more destructive then at some point if we don't get more rational and in control and like you know coordinated we're going to blow ourselves up like holy so this is not a new observation and I think AGI so I think it seems like we don't really disagree with AGI happening so I'm not gonna like give dig deep into like why do I think it's happening or whatever but to me it's the next step in this and this thing like you know why are nuclear weapons you know what are the things training their dangerousness well they're not identic they're not intelligent they can't like go set themselves off still they're still humans in the loop right they're still controls and limitations but that's not necessarily the case when something's intelligent if we have an AGI system let's not have our values and it wants something that we don't want I expected to get that and currently the question of how do you make things want the things we want like I want several things there are many things I want such as me and my friends to have a good time you know and to live happily and free of pain and suffering and so on and we don't know how to get those values into an AGI and so by default we will not get that and you know maybe the agis will then you know you know liquefy the planet blast off into space and make some kind of you know cool cyber Universe I don't know but like probably not probably they'll do just something random that we don't want and that'll just be it I think that's tragic I don't want that to happen thank you very much for that Connor uh Yoshi let's just test your audio levels oh but I just don't bring them up but um let's throw the MacBook Pro microphone so the levels better but unfortunately Riverside when I started it selected the wrong microphone and I cannot switch it during runtime which I didn't know was because I my MacBook was pretty far away from me and now I put it close and put up the volume and I think it should be understandable now good sorry sorry your response show to us about it yes let's get back to this thank you so much Corner um let me take a little bit uh step back so uh there is from the perspective of most conversations one shared Multiverse that we are inhabiting and uh that we cannot actually know what's going on in there but we are making different models using uh different perspectives and different criteria and calibrations of our epistemology using our individual little Minds and uh for this reason I believe that humans have difficulty to play at the league where you would call them generally intelligent and the general intelligence of humanity emerges at the civilization level right it's basically you need a thousand years of a pretty unbroken intellectual tradition before you discover touring complete languages and the criteria on uh whether something is true and what meaning is and so on and so on so many of the quite uh criteria that are necessary for us to discuss about what's the case at all and to understand how words can mean things take a long time to coalesce that individual beings in a single lifetime cannot figure out by themselves and this of course also true for many of the conversations that we're having here on the other hand our civilization is not very coherent right as a civilization we are pretty much like an irresponsible child that is explorative and playful but it does not have a species level regard to duty like the duty 2R on Survival or two life owners and so even individuals might understand their Duty our civilization is too incoherent to act on it and due to many coordination problems that emerge uh that if you were to overcome them for instance by installing a dictatorship that is imposing very strict controls on how we deal with our ecological environment how we deal with the resources on Earth how we orchestrate political interactions between nation-states and individuals and so on we are not completely coherent right if you have a world that is diverse it means that we lose a degree of coordination and in our own mind we do have mechanisms that impose this coherence like our I think despite Tim hating the word Consciousness I think that's the role of Consciousness it's to make us coherent and to fix the word and uh like it Okay the reason why both my hands are part of the same game when I move them and not moving independently and are sometimes working together sometimes not is because there is a single interpretation imposed at some level of the coordination of my mind that makes my mind coherent and that is not the case at the level of our civilization or even our state and uh reason for that not being desirable is because we would lose a lot of Revolution and possibility if we impose that single interpretation and I believe that's also vital in this course that we have this multitude of perspectives because the world so far doesn't fit into a human mind and that's also one of the reasons why I'm so grateful that positions like say early users uh Elise keeps positions exist and Conor is positioned taking a position that is very adjacent to this I feel right and probably also owe some influence to elise's writing over many years and by many people dismiss early yeezer as some weird guy who is not really a proper academic and scientist or whatever I think that his influence is undeniable and his intelligence and well intentioneness is undeniable and uh I really think that he deserves a lot more Acclaim that he gets in the public Arena despite me disagreeing with him and but I think that he is being treated unfairly in that uh the way in which people respond to his arguments do not actually engage with the core of his arguments that is if I take elise's position I'm confronted because most of the counter arguments that are being made towards aliaser I would not be convinced and not because I'm stubborn but because these are just not arguments that engage with the substance of the points that he's making and the nice thing is that corner has been making very similar arguments right and uh I I don't think that they are wrong for an obvious and trevorial reason and I don't think that there should be dismissed but I think there should be extremely careful considered but there are different perspectives and one of these perspectives starts with the general aesthetic that is um what is the wall of Humanity on Earth and what is our immediate future without AI so what is the baseline from which we are interacting and so when we are discussing existential risk if we zoom out far enough we know that the next Super Volcano or the next meteor is going to end Humanity's Reign on Earth right and there is a small possibility that Elon can build a more space that can stay stable for long enough to repopulate the Earth but it's probably not going to work out so uh the over a long enough time span we will be dead we will not sail happily into immortality and uh on the shortly enough time spent you're already facing grave existential risks that are homemade vanity is in a very interesting situation Homo sapiens is no longer the same being that existed uh 20 000 years ago or a hundred thousand years ago we have changed in some very fundamental way when we became first in agricultural species and then a technological species and this technological species has uh transformed itself into something quite different qualitatively similar from a grasshopper to a locust switched within very few hundred years from uh 100 million people or few hundred million people to 8 billion people and this was enabled by a number of technologies that uh do not seem to work for very very long in the same way as they work now right and we might might be possible that we will always be unlocked the next technology that is able to solve our problems uh in the same way as the locals might always find another Forest that they can eat but uh it might also be two that is not the case and so some people started to consider this possibility in Earnest for a long time and the first people who did computer simulations about this was the club of Rome and with the limits to growth and in this study that was taken took place in 1970s there was a prediction that our present civilization is present form is going to go through a population bottleneck in the century and I don't see any evidence in the data that I'm looking at with respect to global warming resource exhaustion and so on that is suggesting anything to the contrary that this prediction is not likely to come true where there is not 100 certainty that it comes to but there is a substantial likelihood that humanity is going through something that is in a population bottleneck uh our civilization is present form is going to collapse and it's going to be replaced by something it is far less comfortable and that is going to lead to a lot of extreme hardship and disaster on the way and there's even a possibility that we are facing an existential risk due to for instance the oceans tipping and the atmosphere becoming unbreathable as a result right these are the issues that some people are really really concerned about and that make a lot of people depressed and when people are faced with existential risks they typically become very uncomfortable and they try not to look at this very much and this is the case with respect to global warming in the 1980s and 90s and even now to some degree it's just more on the agenda because people are seeing it more every day um but the predictions were already strong and the Tipping points were mostly in the last century for for this climate development a similar thing is happening with AI there is a possibility that is seen as more remote and more abstract that uh we will build systems that disrupt our societies and our interaction with the planet in such a way that humanity is going to undergo a dramatic disruption and we can also ask ourselves does it really matter if Humanities goes extinct or if only 90 of all people go extinct well and our civilization ends and gets replaced by something that is far less comfortable right that's uh it's it's not an easy question to answer whether this is such a big difference uh it depends on what you think really matters but I personally think matters is life on Earth that is really really amazing that uh to sell formed on Earth or came to Earth came to be in some sense and populated the Earth with thinking complexity of a sentience with something that is able to defeat entropy for a very long time and if something happens to us then we will not be the last intelligent species on this planet Evolution will go on it's extremely hard to stabilize the planet the cell itself once it is has distributed itself and it changed itself on the planet is extremely robust so I think for as long as there is atmosphere on the planet there will be life on Earth and it's going to be amazing and every species that exists on Earth is either relatively boring and found some relatively boring stable Niche or it does something super exciting for a relatively short time in in terms of Earth's time right and we are one of those species that does something that from a certain perspective is very very exciting even though a lot of species would disagree because they find it very disruptive what we're doing and go extinct and you could also say from a third-person perspective not only that humanity is a very beautiful but a short-lived phenomenon but you could also say that after a certain point we no longer interface very well with our part in life on Earth and sentience on the planet if you're not very good stewards of this planet at the moment we are not good stewards of Gaia of the spirit of life on Earth so to speak and instead what we have become in our technological societies are paper clip maximizers our civilization in its present form is a paperclip maximizer is basically taking the diversity and richness and complexity of what is exists in our planet and its potential and turns it into landfill and uh so per default right now you're dead you're already facing a homemade existential risk and there is nothing built into our society that seems to be able to stop this right so per default our society is headed for crash and collapse and our future looks bleak in this regard so this idea that for default we are going to sail into eternity and uh mutate ourselves into utility monsters that can fill the visible Universe it's never been in the cards this is not who we are we are in extremely smart but very short-sighted monkey species that is playing a very exciting but very short game and in terms of life on Earth and now we are building a new technology in this new technology is changing everything it puts all the bolts back up into the air so there is one consideration that I would like to make and this is AI is opening up the space of possibilities and of what is in our future and currently most of the world lines that we are in for modak can see most of the probabilities end up in disaster and I think if you want to survive as a species if you if Corner cares about his friends and his children and everything in the world that looks human and is accessible to human beings we will need AGI to uh to keep that around there is no other way he cannot survive without AGI thank you so much for those comments uh Joshua and and one one thing we might pick up on slightly later is whether it is indeed a contradiction that you can think of humans as simple utility maximizers yet we're beautiful yet it's yet it's wonderful um Connor your response uh about 10 minutes go ahead if I may first ask you if there's a short question uh clarify which part of what you just said do you think I disagree with if any I did observe you when I made this argument and uh you did not look to me as if you were ever ever put the argument in any way so I also don't think that it's an argument that didn't occur to you uh the main reason I made this argument is because I felt it was missing at the step of the conversation I'm not here to uh to try to tell you that oh I think you didn't think about anything here uh no I think we are here to open up a space of ideas well and so the interaction between these ideas and view both have to hold down our end of the tent well well thank you Cheryl then I'd like to thank you because I think you did a great job in actually bringing usually pretty hard to get into in a conversation but are actually very worthwhile talking to so thank you I feel like you are you know helping earlier the conversation yeah and it's not also the end of my argument of course of course I'm sure there is much more on the other end which is more specific about why I am very hopeful about AGI but uh let's make the next step yeah I would love to get into that as well but it's to start with what you just said so I fully agree with you and this is kind of the argument I was making so by default Humanity ends in ruin this is the default outcome this is the full outcome for any intelligent species that can't coordinate that can't work together they can't become coherent and can't coherently maximize their values whatever those values might be uh this is the default outcome of course you're I really liked how you talked about how general intelligence emerges more on the civilizational scale I think there's a lot of truth to this um I think most people I feel like really underestimate how much of intelligence is not in the brain it's in social networks it's in the environment it's in tools it's in you know culture mimetics Etc like so much of what we consider human is not the brain the brain is part of it of course it's an important part of it but it's like definitely not the whole thing a lot of uh things that we consider normal you know like oh surely every intelligent thing would have this are things that humans invented they were not things that humans were born with you know and we split off from chimps and so on so I think these are really nice points um I also think they are uh I think they're all interesting and they're all nice points but I feel like they are distracting from the core point I'm trying to make here is that I was like I agree with a lot of the high level things like we need to build great technology to enable a great future 100 endorsed absolutely I love technology you know I love computers I love you know things I want I want us to have better medicine and a better coordination Tech and better all these kinds of texts like I I you know I want us to go to the stars and to do wonderful things of course and I agree if Humanity continues on this path they will not get that we are not going to get that if we don't improve and this is this there's a uh the subject you're making here that like coordination is impossible I don't think this is necessarily true I think coordination Tech is just something that our culture happened to undervalue I think there are other timelines in which Humanity early on invested extremely heavily into developing better coordination technology to create to make civilization more coherent and I think this would work um but it's not the not the check tree that our civilization decided to follow I don't think it's the most common Tech Tree I think it's unusual Tech Tree I think the Easy Tech Tree is the you know Industrial Revolution Tech tree that we did follow but I think there are civilizations that could have gone differently um so I don't think that's that's the given but now that we are where we are I agree we we've done this is we've gone very deep down the tech tree for energy and power and not very far down the coordination civilization that the technologies that we invented to create coherence in the society are absolute feudalism which is not compatible with other means of production right now fascism and um theocracy and totally disagree there is uh there might be other technologies that we haven't discovered yet but uh there is a reason why we don't favor those Technologies I completely disagree we have a ton of coordination technologies that do work and they're not Global Solutions but they are a tech tree why don't we use prediction markets for everything right yeah yeah but like you can you can bootstrap to buildable coherence why don't we have better epistemology why don't we have you know better you know enforcement mechanisms laws like there are many things you can do we didn't do them and so you might say oh X is unfeasible and I'm like yes because our civilization did not invest into them it would have been completely possible to build a civilization which much better epistemology it is completely possible to build a human scale civilization not with higher intelligence just human scale with much higher stance rationality with much higher epistemology much better Market mechanisms with much better truth seeking mechanism much better transparency mechanisms with culture and enormous I think that if it was possible it would have happened the world that's not how this is exactly the one possible world but I just disagree then I think we're using the word possible differently I think like I think the when I use the word possible I mean from like eight obviously model perspective is that like I think most of the most history is random not all of it but like a lot of it is random like you know did Archduke Ferdinand have to get assassinated that day if you actually read the thing it's a lot of wild coincidences you like escape from the Assassin the Assassins gave up and then he happened to turn a corner and bump right into them like in my conception of the world didn't have to happen that's kind of what I mean like it didn't have to happen like you didn't have to get assassinated you know like you know life bits didn't have to invent formal logic when he did he could have just like never been born like they're you know this is what I mean when I use the word which might not be how you use the word possible I would have had a sequence of world wars until we have nukes one way or the other a number of reasons uh that gave theoretic reasons and economic reasons and so on and trajectory reasons that led to these developments and they're not deterministic in a sense right there are uh of course there is a fundamental question to which degree the universe is deterministic we are talking about models of the universe that we're constructing that are very coarse grain in which we say that based on these models in these model universes uh what are the degrees of freedom that exist but uh when we look at the world it's made from people and people are not that skilled and epistemologically sound and deterministic unless somebody forces them to and if we build institutions that try to maximize this then also these institutions have a certain degree of capture and of rent seeking and of incompetence and when you look at our best scientific institutions there are unaware of very large parts of bodies of human knowledge that just could not be integrated in The Sciences so far because of various cultural and epistemological and interface reasons which lead to a disconnect between different groups of people in the world and in a sense I don't think that any society in the world has achieved uh the full coherence between uh or people who are smart and have sound epistemology I think we're probably not going to make progress on this point um but that's not the point Phoebe it's not necessary that we achieve coherence what's necessary is that we achieve a space in which we look at the different perspectives in an adequate way right so obviously defined when we agree to disagree and you say if people hadn't make the following five stupid mistakes or art history and instead people would have been not accidentally unreasonable we would have converged to a coherent society and I don't think that's the case well that is a society made from people uh is as they are with their small monkey brains and so on it's surprising that achieve this degree of coherence that's fine I expect I will not be able to convince you expect your models are pretty deeply ingrained and you have some pretty fatalistic models there's a possibility that you're right but I don't see you making a proof that you're right so yes and so I I just want to add the missing perspective here fair enough I think this is really fair I'm not trying to make a proof here I would I was just kind of stating some of my beliefs before I move on to my actual point yeah like I was just saying as a side note which I will not further substantiate to this point my models of reality are just way more unstable I think there are way more ways how things can go I think people have way more control over reality than they think they do I think even small numbers of people have way more control over reality than culture wants us to believe us that we do I think this is various instrumental reasons why this is that culture is supposed like if you have very agentic people this is usually very dangerous and they're very unstable this results in unstable societies I think this is true I think my model again you know we can talk about this more later maybe but I'd like to actually make my previous point but just to like end at that point my model is is that there's kind of like this like swamp of low performance between you know fully heuristic driven versus actually you know reflexively rational and in the middle like a little bit of rationality is worth the no ration is then worse than no rationality and so this creates a national evolutionary trough where even though the the fully enlightened rational solution is more powerful getting there is so expensive that it doesn't happen very often it's very expensive it's very unlikely to happen and in fact Evolution will favor mutations that go to go there so but yeah I I do think that you could have very Advanced cultural technology that allows people to be very rational to be very coherent not infinitely so of course not you know but like I I don't think we're at the limit of what is possible in coordination that being said I do think that we're not only not a limit we're very far from it I think if we had more than NF1 examples of societies of your real world history a thousand times you know most of them would look pretty similar to what we are I do agree with that but I don't think all of them would and okay but now we're in this we're in the timeline we're in No Escape so what I care about is okay we're in the timeline you're right coordination is hard it's really hard it's a lot of problems here if we just you know keep going like this for another 50 000 years like a super volcano will destroy us totally agreed but then I have to jump back to the Practical level like all right give me 200 years to do alignment research not 50 000. give me give me 20 years it could be 20 years is that a deal we can make so like a lot of the ways I think about how you know how to reasonably think about things as trade I think we should be we should be trading on things I'm like people you know someone comes up to me and he's like oh look you know I really think we should you know go to the you know go to space as fast as possible it's the best thing for Humanity I'm like okay seems reasonable well I also see all these risks or whatever you know they seem like much bigger problem so if you care about you know your life spreading to the universe and I asking if you if I asked you for okay we never go to space well yeah then you should say like well no that's a terrible deal this is not what I would be asking for what I'm advocating for is okay let's give it like a few more decades before we you know build the Uber you know cyber Von Newman Galaxy machine Galaxy eater and then if after you know like another 50 years or 100 years or 200 years or something you know of society progressing technology progressing philosophy progressing all of this and then we build the Galaxy eater this is what I'm saying I'm not saying never build you know a Galaxy eater I'm saying you know let's first make sure it's one that we like that actually does progress life so we cannot give you that I can give you this I can pause my own research for a few decades or permanently but Society cannot give you this what uh the uh EU government or can give you is regulation that up 80 percent of AI research for a time uh right or permanently and visit Europe but it's or visit parts of Europe that submit to it or the U.S government can give you a regulation that establishes a permanent Monopoly for open Ai and Google on the existing language models because the others cannot afford to go to the certification process but I don't think that you can impose a ban on compute and developing new algorithms that lead to a self-improving agency in the near future and unless you uh basically create some Global institution that is regulating access to gpus or to compute at scale and uh so there's this B doesn't exist and uh it's possible that that there is a small possibility that this can be done but I also suspect that it might not be desirable because the uh people that are already of Institutions that are currently in a position to impose that regulation are acting not under incentives that would lead to regulation that is actively actively preventing this I think it was very well illustrated by the letter that was written by fli Under the this institution under the initiative Max techmark and Anthony aguir instigated by a alligator to impose a pause on uh training large language models and uh this was uh comically signed by Emet who was uh the CEO of stability AI who did not intend to uh to do this was mostly thought as a public statement and Elon Musk who at the same time was setting up his own tools GPT uh initiative and so it's also not intending to implement any kind of pause and the majority of the people who say this are people who are on the record of saying that they believe that egi is a fantasy and a science fiction idea and everybody who was actually worried about it is a bad person because it distracts from the societal risks that are enforced by language models uh uh have exhibiting biases or disrupting labor markets and so the reason why most of the people signed this letter was not because they were concerned about an existential risk so self-organizing AI taking off and destroying the planet for human life or making it possible for it for us to coexist with it in the future but there were concerned about very other political things and the regulation that would have been passed by them and even if the regulation would have applied to large language models would not have had the intended effect because if you tell people you can no longer use your existing GPU forms to uh train Transformers on large amounts of data without institutional oversight that delays everything and makes it similar to say um cloning people or uh stem cell technology and so on that are very difficult to perform right now um right this would basically get people to develop new algorithms is that uh CPUs which is my paradoxically speed up the developments that early users are worried about and so I think that regulation makes sense only when The Regulators understand the situation that they agree on and they're globally aligned so you don't get a prisoners dilemma situation so a principal actor problems or the issue that your Regulators get captured very very early on by completely different interests that hijack the regulation to achieve goals that are very different from the ones that you want to achieve right there a lot of people which want to protect existing industries from emerging technologies that we actually all want to have because most of us agree that uh the creative industries that we have don't agree as employment programs they exist to produce certain cultural artifacts and it's beneficial for us if we can make more of these artifacts in better ways and if we want to have a less disruptive way in which this transition takes place Regulators are being asked to do this but it's a very different way uh argument in a different Arena than existential risks that we are specifically talking about existential risk is a topic that extremely few people right now in terms of AI take seriously so we are already on a very long tail of a distribution of opinions and perspectives of the world in which our own discussion takes place and I agree with you that it's an extremely important one because existential risk is very very important and debate in which the arguments against existential have to be diffused is complex and more complex then I see most of the discussion happening but I I don't agree that we're already in a situation where we can ask the public or The Regulators to impose regulation and hope that they will achieve what you want and not do the opposite I believe that there is a chance and it's something hopeful with respect to AGI that for instance is a scenario in which we can achieve non-violent coherence in a society right and one of the main reasons by most of the ways of which we would achieve more coherence like for instance in covet our institutions fail to coordinate in China they didn't but the result of the measures that were imposed on China was that the population went on the street because they were no longer willing to subject to the draconic measures and the help visited implementation that was necessary as a side effect of this centralized imposition of the zero covet policy right as a wizard you had a catastrophic failure of the zero covet policy and a very large number of people were going into the emergency rooms and died as a result because nobody was vaccinated right so in some sense neither of these Solutions were optimal but imagine you give everybody their own AGI and you don't make the AGI authentic but you make an extension of yourself in a similar way as you say well from alpha as an extension of Stephen Wolfram but with the way that is much more useful to everyone and it allows everyone to have facing their own Google and to model the world as toolsfully as mathematically possible and that thing is your own disposal and it's not serving some cooperation that is forcing their technology down your throat but that is serving you as every individual being I think this was dramatically level the playing field and it basically it makes it very difficult to lie to yourself about the impact of your actions and for institutions to lie about each other about the impact of their actions and it would allow us to coordinate and force us to coordinate with a much much higher degree of coherence and this is a possible scenario where in AGI development might indeed help us to make a big turnaround in many of the developments that threaten our survival or the Integrity of our civilization new term yes I mean yeah yeah you're correct if everyone had a pocket HDI which is fully aligned with human values which is epistemologically you know extremely coherent which does not optimize for things we don't want which is like deeply reflectively embedded into our own reasoning and our thinking yes that would be good but that doesn't happen by Magic you have to actually do that someone has to actually figure out how to do that someone has to actually develop the technology to do that and they have to actually deploy that and they have to do it without some other some iak on Twitter hooking it up with the utility function of maximize the entropy of the universe if you don't do that you die yosha you die you don't get this nice outcome for free you get this outcome if you coordinate it if you work hard and if you solve a very hard technical problems that do not get solved by default they can be solved everything you just described is completely feasible these are actual things that our physics allows to happen in but they don't happen by Magic the universe it does it isn't kind we're not the main characters there is no rule that we have to make it you you talk about all these fatalistic things about how you know coordination will never work oh we're always you know pushed down this you know track of industrialization or whatever why would it be any different here why would it be any different why wouldn't it just lead us down the darkest path possible by default and that's what will happen by default if your model is that people do not have control that it is not possible to steer away from the global minimum then yes we are and you should go spend time with your family until it's over you should not care you should give up if you do think that there are other paths that are accessible that there are actions that people can take if there are coordinations that are possible to get to the kinds of scenarios you describe then we have to actually do that it does not happen for free and this is what I'm advocating for I'm not advocating this is easy I'm not saying this will happen I'm not saying oh don't worry the government will fix it I'm saying the opposite everything you say I'm like yeah true like you're saying oh this video game is really hard yeah the enemies have really got hit points I'm like yep and they have you know dangerous attacks yep true and if you get to the end there's a really hard boss yep true so you shouldn't play the game and I'm like look the default outcome is you lose the default outcome is you lose the default outcome is entropy wins some random AGI with some random ass values that does not caracapa Cosmopolitan life on Earth you know you know wins over or maybe it's a bunch of them and then they all you know coordinate because they can actually coordinate because they're actually coherent because they are actually super intelligence so they can coordinate Against Humanity and then that's just it and it's just game over forever okay okay we have half an hour left folks so I've put a call out to ask some really good questions on the live comments the live comments are extremely spicy we've got a lot of I think we've got 400 concurrent viewers at the moment so this in the meantime can I simply respond a little bit to Conor because I think that we have you may quickly misunderstanding that is important for our conversation um I think that my own perspective is very specific it's a one that where I choose my own Aesthetics and my path through the world and the people that interact with the projects that I work on the bets that I make if that's inevitable in this own perspective is normative I have an idea of how the world should be from this perspective but when I want to understand how the world is and what's going to happen I cannot take this perspective I need to take all the perspectives and so basically there's an equality of opinions and it's not a normative thing that I say it should be a state in which we are uncoordinated or should be coordinated I'm just trying to understand how the world is in this regard and so it basically opinions I like mushrooms they grow wherever their space for them and the climate is right and they will co-exist and uh so uh I'm not upset that I exists I'm I just think that you could oppress eak by censoring it or canceling everybody who has this opinion on social media but this would not invalidate the position or the perspective that they're bringing where they look at a certain possibility and you think uh it's not the most likely out current I might agree but uh it's uh that is not the point where the point is that there is a variety of positions uh that we can take and my own position is the best possible outcome is that we are co-existing with the global planetary Consciousness that is basically identifying with the regulation of uh complexity on Earth and that is interacting with us because it is uh not because it is adopting human values because I don't think that human values are an extremely useful concept that actually exists in the form that we can specify and formalize and that people would actually agree on they are more like a useful fiction that we use in context where we already have established shared Aesthetics or uh we have already constraints that uh imposed on our behavior in such a way that this does not really play a big role if you don't really have an agreement on what human values are and how to align them and if we take this as a measure the way in which humans basically arrive at their values right now and how they relate to them when they choose their actions and implement this in AI or emulate AI it's going to lead to disaster I guess we agree on this right so this notion of alignment with human values is I don't think that's something that works at this level of granularity of discourse that we have right now or also I I would disagree of being aligned by anybody else right I'm an autonomous agent I have mobile agency which means I determine what's right and wrong I don't let people around me determine whether I'm going to be a communist or a fascist or a Social Democrat that's not for you to decide that's not for society you decide I decide my values and I if about the system that is smarter and more Lucid than I am I don't think that I can align it it will align it with what it is and what it can be and so for me the main focus is that we need to if we take this possibility into account that we will share the planet with systems that are smarter and more Lucid than us that understand the world better than us in the relationship to the universe that they have and they can have better than us the way we co-exists with them is not by imposing human values on them via reinforcement learning with human feedback or some other trick because that's not going to work it's going to be aligned itself with us if it wants to if it lacks us if it loves us so uh and from my perspective the thing that is most important for us is that it values and understands our own mode of existence which is we are conscious beings what's important for us is not that we have two legs what's important for us is that we are conscious autonomous self-actualizing agents that want to realize particular kinds of interactions in Long games in the world and I think this is the thing that we might have to offer to an AI that is negotiating with us about our own Survival and I think that requires a big cultural Vibe Swift to have this conversation at all and I would like to open a space for this thank you very much so we'll go to audience questions now actually today at 60 seconds 60 seconds you may go on 60 Seconds I'm just gonna just gonna respond to all that I can't respond to all that because we just opened a huge can of worms but human values and Alignment we could go on for a gate so it is it is in many forms yes yeah it is a departure session I'm not trying to say it's not worth it I'm just saying can't do it right now so shelving that for the moment I think a lot of what you're saying is that you're looking approaching things from an epistemological perspective you're describing things how they are I think this is fair enough I think when I go into discussions like this I think the epistemology is important but I I ultimately in coming things from a decision theoretic perspective I care about winning I care about things going well I care about discussing how do we act so that things go well uh if stomology is necessary groundwork for making correct plans and acting correctly to achieve the goals you're trying to achieve but it is different you can be very very correct but very valid planning you could also be very good at planning but very incorrect about reality so I think I agree with a lot of things you say about reality I think where we're disagreeing is potentially to some degree on our values the thing we're trying to optimize for and how we get the things that we're trying to optimize for and I'm mostly interested in talking about that but we can talk about we can also shelve that for a later discussion so audience questions okay so the the first one uh from Jonas and this is to you yosha how did Minsky and I know you were very good friends with Minsky approach AI alignment especially his concept of negative expertise I'm afraid I cannot really say this uh I'm not competent in this also I don't think that I was good friends with Minsky I had the honor of visiting him quite a few times in the last year of his life and um so uh I was part of his salons and had a number of conversations with him and he himself did not think that EGA was imminent because he did not believe that deep learning was the right way to achieve it from his own perspective deep learning was a distraction from AI and um he felt that it had been no progress in the last 15 years and that's uh that's a quite different perspective right and if you look at the implications of the Society of Mind he saw the mind as an interaction of agents that had a degree of autonomy and that at the same time have to achieve coherence he did not have the perspective that this is what the mechanisms of achieving that coherence is he mostly looked at for instance structures of management that emerged K lines which I think can be translated into something like Dynamic attention heads in a continuously operating Transformer if you want to um maybe this is the wrong perspective but I think it is a possibility to see it like this you could also think about this as something that happens at the level of society I guess that for Minsky's perspective existing in an evolutionary environment and in the same way as the contrails of jet planes are nature's way of putting this particular kind of cloud into the world humans are Natures of a to make the Rocks think and impose uh Agency on non-biological parts of the planet and there are evolutionary outcomes of this and we can sit back with big eyes and see what's going to happen and it's really really fascinating but our agency of this is limited right if you if you basically take out your individual hoopers as an activist for a moment and realize that if you are not this activist there might be somebody else taking your place and the fact that you are corny or Amigos is incidental right there is of course somebody like this has to exist and we happen to be it and all the others exist too and so if we if we disengage for what we do what we think what we want for a moment and just look out at what's going to happen the the landscape if that changes that means that our own activism is distorting our ability to make sense of the world and so if we think about what should be done in a world where we don't really able to estimate all the outcomes of our actions we might systematically impose biases in our own modeling That clouds our vision so I think both is important it's important to once you make a decision and make bets to become an activist and say I want to build a Consciousness research direction that allows us to build agis that can interact with us empathetically and that we can interface with and such a level that the AGI would have a chance to want to coexist with us if we built it and we cannot avoid such systems being built which probably is the case and your perspective is a slightly different one and I think they're both valuable but let's not log in too early on one of those perspectives and lose the entire space of possibilities okay thank you very much for that yosha a question of you Connor from hey hey um since when did humans have good values and are they worth saving what is this good you describe where exactly does it come from oh right you're a human aren't you weird are you going to add any more to that I this is one of those questions where the correct answer is just like like either you would have to have a like days weeks long deep dive into like psychology philosophy or the correct answer is just kind of like like yeah humans have bad values that we don't agree on humans aren't consistent blah blah blah but also like where else do you think value in the universe comes from like and like even if you disagree if you think value in the universe comes from like somewhere else I have things that I like and that a lot of other people like and we are willing to coordinate on to have more of I like people having nice lives I don't like people having pain and you know if we have a few beers or whatever we can have a long deep discussion about like the true metaphysics of it all but ultimately it doesn't matter ultimately there is no it does it's really important because it's really the question that decides which direction you want to go into if you cannot justify your own values you cannot interface with people who have justifications for their own values it's arbitrary there are psychopaths okay they are human there are Christians who are human uh the our identity theorists who are human there are communists that are human and humanity is just beautiful and uh we should all uh make sure that the AGI is having human values this is not the battle of the philosophy I think this question that we supposed to you is bio important and you cannot just dismiss it uh with making a pun to turn I mean to be clear I can actually do that this is because this is as this is as valid of a philosophy as yours it's like there is no underlying ground truth everything is chaos the universe is nothing it's all nihilists I'm right you know moral realism is a joke like it's everything emerges from you know bargaining and from values and from Mis optimization stuff like and like sharding and so on like we can talk about this if you want to but this is not the place to have this conversation and like so everything has to add up to normality this is the thing I think a lot of philosophers forget is that everything generally adds up to normality after you go down deeply you talk about a causal decision theoretic bargaining Solutions and whatever you can go down you can get all these justifications whatever but eventually you're going to pop back up to oh I don't like pain you know you can just I can justify this using a causal bargaining theory if you want me to but okay let's let's suppose I think we could spend two hours talking about the the more real more realism um Joshua a question to you and this is this is actually a pretty intriguing question do you think that AGI could be or maybe the last philosophical invention of mankind of course yes I mean that's the whole idea that's uh the reason why lightness was so fascinated by it when uh I was introduced first the philosophy teacher said that slightness had this weird idea that somehow we would translate every argument into numbers and then perform a computation that figures out what the right resolution of the argument is and uh the philosophical Professor uh thought this was a Preposterous idea and every student in this first semester completely understood why it's a Preposterous idea and it's very hard to to see by somebody like that this was regular regarded the smart and such a stupid idea but uh this is exactly the idea of AI it's the idea of mathematization of the mind and the current iteration of this is that we translate the argument in two numbers by constructing an embedding space and functions that allow inference in that space and then we construct a new number that keeps the outcome and currently we are thinking about ways to build epistemology in the systems and make them more coherent and allow them to combine empiricism with reasoning from first principles so it's really the most exciting philosophical project there is in my view born on and if we succeed in this of course it's going to be better at philosophy than us okay fantastic a slightly more down to earth question for for you Connor do you think that open AI is still increasing the power of gpt4 not through training larger models but through other methods and perhaps you might have some background knowledge on what those other methods are I mean yeah obviously like obviously they're making it better by any methods they can come up with I heard rumors and leaks about several things but you know like most the things you'd expect for the most part you know agents memory context length more training data bigger models sparsity just all the usual stuff like nothing I haven't heard anything that I found particularly shocking or surprising just well you would like if I was running an AGI lab and I was just trying to you know speed run you know omnicide as fast as possible it's just the same kind of things I would do okay okay fine um you're sure um this is a question from APO uh you seem to have a deterministic stance on geopolitics uh you made some comments about World War one and Technical development and the question therefore is do you see AI alignment as taking a similar predetermined route and there was another question we can tack on from someone else which is to what extent is the um understanding of Free Will something that might separate your arguments that's philosophically uh extremely deep and important question but and I'm I'm afraid we cannot do it justice but uh very briefly I think that Free Will is the representation of an agent that it makes a decision for the first time under conditions of uncertainty right this means that you cannot predict that decision that you're going to make because in order to know what this decision will be you have to run the universe to that point and so you are working basically at a knowledge Frontier of the universe and it doesn't really matter whether the universe is implemented stochastically or deterministically the uh everything that is control structure is the result of the deterministic parts the stochastic parts are just deleting bits that you computed before and introduce noise and but you don't get more free will if the universe is randomly forcing you this dice froze to do things then when the universe is forcing you by this deterministic loss to do things but the way in which we conceptualize ourselves is that we are systems that are made of subsystems that are Loosely coupled from each other and we treat these subsystems as objects as individual functions that interact with each other that's how we get causality and we don't then you think about the world interact with the world as it is we don't interact is the state Vector that is describes the evolution of the universe we just interact with the vein which we think about the world and the way in which we think about the world uh history is not fully deterministic but history is following certain constraints which means there are certain things which are more or less likely at a given time and that follow from the outcomes of certain situations right if you if a society becomes unsustainable it's running into a revolution and you can try to identify the criteria for this and the time frame for this but when you look at down at the level at which uh particles interact that's a very different granularity and at this level you will see that this is panning out no matter what right and it's people get very confused when they try to conflate these different levels of description but uh Free Will does not exist at the level of where you describe the world as particles it only describes is is that where you describe the world is the level at the level of interacting people and when you talk about this you're always talking about these Frameworks about these systems in which you model the world thank you Joshua um okay I've got a question in from Dr Duggar the one and only he says there to you Connor what is currently the most promising path the state of the art mechanics if you will for achieving aligned AGI God I wish I knew and I think we have any current solution which actually would like actually lead to unlined AGI there is no I think there are people who think they have some and you know maybe if I'm totally wrong about how reality works and like they get a hundred years to work on it it could work you know like maybe like John Wentworth stuff um he works on like you know selection theorems and like natural abstraction hypothesis we said that stuff maybe something like Vanessa coso is like infra bayesianism pre-dca type Insanity like um you know I don't even think my current research will lead to aligned AI not AGI I I don't think Mike's my Solutions are nearly like will lead to that we need to come up with something better um yeah I think yeah a stupid answer but I don't think any current thing will work okay thank you Connor um to you Joshua uh do you think merging with AGI is the evolutionarily favored path for most sentient species in the universe and and why might you think that I think it's in relationship to uh what chronologists that I observe in my own family where we have very different personalities how we learn to align with each other and when I want to achieve alignment with my children how to do this or we can also look at the way in which species that can align with people like for instance dogs and cats align with us right the interesting thing about dogs and cats is that they don't align because they are fully coerced or because we can program them but they are to a large degree autonomous and they relate to us in a particular way that from the local perspective is rational in the sense that it leads to a stable equilibrium it's not when they're smarter that they realize oh I was wrong to align with this and it's also true in a sustainable relationship with other people that they don't align with you because they are just not far enough yet to move out and rebel against you but because they are actually incentivized to do so objectively in some sense it's like it's a win-win situation if we align and so uh this win-win situation typically happens because we are invested in a shared system in shared purposes and I think we need to build uh theories about these shared purposes and which conditions do shared purposes objectively exist the systems are set up that they lead to stable equilibria uh in the sense that we don't end up in a gradient that gets out of this attractor where the systems stay in alignment and so I think we basically would need to develop theories about having shared Purpose with non-human systems and some confusion exists because of the nature of these systems I don't think that we will live next to AIS in the future we live inside of them right there will be systems that are permeating everything else and from this perspective merging this AGI is already happening when you are joining a company or when you're joining a nation state or a civilization you are in some sense merged with an artificial general intelligence that instance artificial that is being created by intentional actions of people and some of this is going to perform computations outside of biological brains and it's already happening and they can be also in the future which is not happening at scale autonomy outside of biological brains and in this case I still think that we can have shared purposes uh when we think about merging I do think that there is a trajectory that is conceivable in which the end game of egi is that it understands itself and how computation Works in general and it's going to virtualize itself into all substrates including biological organisms bodies and brains and ecosystems so we will find integrated agency in all substrates but is this agency going to be fundamentally different than the agency that runs on this substrate is it possible to implement something on your brain that is fundamentally much much more efficient and better than the stuff that already runs on your brain or is this just going to be more aligned with each other right and I suspect it's going to be the latter because there's already a search process running on your brain for looking for the best possible organization and it has been done for over many generations so this is more like you need another agent on the same substrate and you are going to a merge with it you're going to negotiate with it and the outcome should be that the resulting agent is at least this goes as a better one of you and uh right so I think in this sense it's conceivable that we all become like conscious thoughts in the same mind if we increase that integration functionally thank you so much yosha and I think maybe final question for you Connor now I actually um put your question back to you again schmidhuber the other day you um coined this thought experiment of what would happen if we had a hundred Von neumanns a thousand Von neumanns 10 000 Von Neumann's what were the the scaling Dynamics look like and and also I mean schmidhuber he's really really good at things like recursive self-improvements and go do machines and you know meta learning right so these things have limits don't they like when you even when you reflect in a language model when you recursively self-improve there are limits so why do you believe that there are essentially no limits I don't believe that there are obviously I didn't mean to put words in your mouth but um is there a gap between you believe that the limits are high enough to cause catastrophe yeah I mean of course I think so I think there's two aspects to this there's one is how much do I think intelligence scales Beyond human I think humans are like like as Josh already said like general intelligence is like more of a civilizational you know scale thing and like barely a human thing and like humans have become more sentient and like more generally intelligent by like memetic and cultural development and so on and I expect a lot of the a lot of recursive self-improvement is going to look like that this cannot be magic it's going to just be developing better scientific methods definitely better epistemology more rational thoughts inside of the agi's Mind more you know structured thinking better use of memory and so on I think there's massive low hanging fruits here like if you put a modern well-trained you know yosha on a epistemological task versus you know you know our cave ancestors from 10 000 years ago even the cave ancestor has the same size brain as yosha you're just going to destroy him at any like relevant you know like epistemological task you can possibly you know come up with and uh a lot of it will look like that then some of it is going to look like that they can you know develop better hardware and better software systems and just bearing architectures very very quickly you know the you know the brain can't really change its architecture very quickly there's size limitations you know the human brain is basically limited by size by the human pelvis and like birth and like you know like cooling is a huge thing like brain the brain actually has a lot of human brain specifically has a lot of like like unique things are like only human brains that are mostly around cooling um and there's also like you know limitation on how much energy our guts can absorb we can't just like eat a bunch of plutonium and take in 10 kilowatts of energy unfortunately um so these are limitations that you just kind of get around um you also get you know instantly for free because the digital system you get things that can perfectly clone themselves that can you know do parallel learning that can do you know that can coordinate amongst each other very effectively that can you know this this gets lots of things for free and then on the other side I don't think the world is very stable I think the world is quite unstable I don't think you need something that is infinitely intelligent to take over the world I think you need something probably only modestly more intelligent than a human like not more than 10 to the power of three one jean Von Newman's probably 10 to the power of one is more than enough if I had to guess um I think there is a limit you know at some point you run out of energy and like you know just like the the speed of light becomes an issue there are limitations to physics there is the landauer limit there are limits to computation I just expect those limits are like you know just like billions and billions and trillions of times beyond what will be relevant like we're we're not going to see those limits like humans are going to be gone way before we get to like you know 10 to the power of like you know six or ten jean-vonne Newman's wonderful Okay so we've got about three minutes left um I just want to take this as an opportunity to thank you both so much for this stream it's been absolutely wonderful and would you both mind just making just roughly one minute closing statements and then we'll stop the stream but thank you so much both for coming okay out I don't mind go ahead okay so I I think the NBN absolutely fascinating point in history and I'm very grateful of having been born at this point so I can experience this which to me is one of the most fascinating things that Humanity can experience during its run and I think that it's unclear what the future is going to be like but I am quite excited for it uh there are many things about life on earth that is very sad I don't think that the default state of organisms is they live in comfort and diverse dignity and that we had this for a few generations and probably have a few more is uh very very rare for an organism on Earth that can experience Consciousness and at the same time I don't know if if there is not more agency than the one that we can see at our time scales on this planet I don't know if for Norman is really the Prototype of the most intelligent being on Earth I think that there might be a spirit of life on Earth uh that is uh integrating in at some level of thinking over uh what happens on this planet to realize that life on Earth is not about humans it's about life on Earth and when we extend this the systems um or when we become the facilitators of that transition which probably will happen at some point in the evolution of life on Earth that the life on earth makes the non-living parts of the world think too then this is going to unlock the next level of evolution on Earth and this is probably something that is very similar disruptive to the great oxygenation event that enabled really a new stage of evolution and a lot of the fungi and alga that existed before this point might not have liked that even though that didn't go extinct but uh of course they were reduced and many species disappeared but it enabled plants and animals to exist on the planet and I think if we were to build AGI it's going to enable things that are much more dramatically interesting than even plants and animals and we might be able to coexist with it even and that is also an amazing Prospect but the whole thing that this evolution is happening regardless of whether we want it or not whether Corner tries to uh build activist alignment and tries to impose regulation against it's probably going to happen and if we are not going to trigger it could be that Humanity goes extinct before we are able to do this and it's being left to the next technological species of the planet to try again but it's probably going to happen at some point this seems to be the way Evolution goes and uh I I think it's great Prospect for Consciousness in the universe thank you so much and Connor yeah so I also want to thank you uh thanks Tim for organizing this and thanks Josh for taking the time it's very fun I really enjoy it we could go on for hours and hours and hours obviously you know barely scratch the surface here like you know in another life you sure would have been my favorite you know mid-centuries 28 it's with 20th centuries German philosopher uh if you had been born a century earlier uh it's great stuff I love it I really enjoy it but uh like again I I care about actual practical things for the most part philosophy is for me entertainment's more so than it is actual things that you know actually need to get done so the the thing I would end on is It's Not Over I don't believe in this fate in this kind in this way I don't think that like this is a healthy morality for a decision-making agent to have I think a decision-making agent should actually make decisions and actually try to try to lead to an outcome that they like I think we can get to the outcomes that yosha likes because those are outcomes I like too you know like living side by side with you know beautiful nice AGI wow that would be awesome unfortunately I just think you don't get there by default this is not the default trajectory this is a very non-standard trajectory for the universe to go down this is a very very very non-standard thing to happen it is possible and I would like us to get go down that non-standard directory I think it's possible I think it's it is possible I'm not saying it's easy I'm not saying I'm going to succeed I expect not to succeed I expect no one to succeed I expect that yes by default the universe just goes down the you know most likely paths it doesn't mean others don't exist so that they're not accessible and so I would like to focus on actually doing the work and getting to those things but anyway I want to live in if I want to live in a future where there can be Yosh you know writing this wonderful Philosophy for centuries and I can read all of it wonderful okay um it's been absolutely amazing to both of you now we have another live stream tomorrow with Connor and Grady Butch uh Grady booch gave the touring lecture I think about 12 years ago and is a real heavyweight in the software engineering and computer science world so um we've got that to look forward to tomorrow but thank you all so much for joining the stream I really appreciate it and I hope to um welcome you back on the show as well very very soon yosha thank you so much for coming thank you very much for facilitating this

Channel: Machine Learning Street Talk

Views: 15,701

Keywords:

Id: Pe3QrrHPmiM

Length: 89min 35sec (5375 seconds)

Published: Mon Jun 19 2023