Donald Hoffman: Reality is an Illusion - How Evolution Hid the Truth | Lex Fridman Podcast #293

whatever reality is it's not what you see what you see is is just an adaptive fiction the following is a conversation with donald hoffman professor of cognitive sciences at uc irvine focusing his research on evolutionary psychology visual perception and consciousness he's the author of over 120 scientific papers on these topics and his most recent book titled the case against reality why evolution hid the truth from our eyes i think some of the most interesting ideas in this world like those of donald hoffman's attempt to shake the foundation of our understanding of reality and thus they take a long time to internalize deeply so proceed with caution questioning the fabric of reality can lead you to either madness or the truth and the funny thing is you won't know which is which this is the lex friedman podcast to support it please check out our sponsors in the description and now dear friends here's donald hoffman in your book the case against reality why evolution hid the truth from our eyes you make the bold claim that the world we see with our eyes is not real it's not even an abstraction of objective reality it is completely detached from uh objective reality can you explain this idea right so this is a theorem from evolution by natural selection so the technical question that i and my team asked was what is the probability that natural selection would shape sensory systems to see true properties of objective reality and to our surprise we found that the answer is precisely zero except for one one kind of structure we can go into if you want to but for for any generic structure that you might think the world might have a total order a topology metric the probability is precisely zero that natural selection would shape any sensory system of any organism to see any aspect of objective reality so in that sense uh what we're seeing is what we need to see to stay alive long enough to reproduce so in other words we're seeing what we need to guide adaptive behavior full stop so the evolutionary process the process that took us from the origin of life on earth to the humans that we are today that process does not maximize for truth to maximizes for fitness as you say fitness beats truth and fitness does not have to be connected to truth is the claim and that's where you have an approach towards zero of probability that we have evolved human cognition human consciousness whatever it is the magic that makes our mind work evolved not for its ability to see the truth of reality but its ability to survive in the environment that's exactly right so most of us intuitively think that surely the way that evolution will make our senses more fit is to make them tell us more truths or at least the truths we need to know about objective reality the truths we need in our niche that's the standard view and it was the view i took i mean that's sort of what we were taught or just even assume it's just sort of like the intelligent assumption that we would all make but we don't have to just wave our hands the evolution of natural selection is a mathematically precise theory uh john maynard smith in the 70s created evolutionary game theory and we have evolutionary graph theory and even genetic algorithms that we can use to study this and so we don't have to wave our hands it's it's a matter of theorem and proof and or simulation before you get the theorems and proofs and a couple graduate students of mine justin mark and brian marion did some wonderful simulations that tipped me off that there was something going on here and then i went to a mathematician chaitan prakash and manish singh and some other friends of mine chris fields and but chaitan was the real mathematician in behind all this and he's proved several theorems that uniformly indicate that with one exception which has to do with probability measures um there's no uh the probability of zero the the reason there's an exception for probability measures so-called sigma algebras or or um sigmative classes is that for any scientific theory uh there is the assumption that that needs to be made that the whatever structure the whatever probabilistic structure the world may have is not unrelated to the probabilistic structure of our perceptions if they were completely unrelated then no science would be possible so and so this is technically the the map from reality to our senses has to be a so-called measurable map has to preserve sigma algebras but that means it could be infinite to one and it could collapse all sorts of event information but other than that there's there's no requirement in standard evolutionary theory for fitness payoff functions for example to preserve any specific structures of objective reality so you can ask the technical question this is one of the avenues we took um if you look at all the fitness payoffs from whatever world structure you might want to imagine so a world with say a total order on it so it's got end states and they're totally ordered and then you can have a set of maps from that world into a set of payoffs say from zero to a thousand or whatever you want your payoffs to be and you can just literally count all the payoff functions and just do the combinatorics and count them then you can ask a precise question how many of those payoff functions preserve the total order if that's what you live or how many preserve the topology and you just count them and divide so so the number that are homomorphisms versus the total number and then take the limit as the number of states in the world and the number of payoff values goes very large and when you do that you get zero every time okay you've there's a million things to ask here but first of all just in case uh people are not familiar with your work let's sort of linger on the big bold statement here which is the thing we see with our eyes is not some kind of limited window into reality it is completely detached from reality likely completely detached from reality you're saying 100 likely okay so none of this is real in the way we think is real in the way we have this intuition there's um like this table is some kind of abstraction but underneath it all there's atoms and there's an entire century of physics that describes the functioning of those atoms and the quarks that make them up there's many nobel prizes about particles and fields and all that kind of stuff that uh slowly builds up to something that's perceivable to us both with our eyes with our different senses as this table then there's also ideas of chemistry that over layers of abstraction from dna to embryos the cells that make the human body so all of that is not real it's a real experience and it's a real adaptive set of perceptions so it's an adaptive set of perceptions full stop we want to think perceptions are real so so their perceptions are real as perceptions right they are we are having our perceptions but we've assumed that there's uh a pretty tight relationship between our perceptions and reality if i look up and see the moon then there is something that exists in space and time that uh matches um what i perceive and all i'm saying is that if you take evolution by natural selection seriously then that is precluded that our perceptions are there they're there to guide adaptive behavior full stop they're not there to show you the truth in fact the way i think about it is they're there to hide the truth because the truth is too complicated it's just like if you're trying to you know use your laptop to write an email right what you're doing is toggling voltages in the computer but good luck trying to do it that way that's the reason why we have a user interface is because we don't want to know that quote unquote truth the diodes and resistors all that that terrible hardware if you had to know all that truth it would your friends wouldn't hear from you so you so what evolution gave us was perceptions that guide adaptive behavior and part of that process it turns out means hiding the truth and giving you eye candy so what's the difference between hiding the truth and forming abstractions uh layers upon layers of abstractions over these of our low level voltages and transistors and chips and programming languages from assembly to python that then leads you to be able to have an interface like chrome where you open up another set of javascript and html uh programming languages that lead you to have a graphical user interface on which you can then send your friends an email is that completely detached from the zeros and ones that are firing away inside the computer it's not of course when i talk about the user interface on your desktop um there's this whole sophisticated backstory to it right that that the hardware and the software that's allowing that to happen evolution doesn't tell us the backstory right so the theory of evolution is not going to be adequate to tell you what is that backstory it's going to say that whatever reality is and that's the interesting thing it says whatever reality is you don't see it you see a user interface but it doesn't tell you what that user interface is how it's built right now we can we can try to look at certain aspects of the interface but already we're going to look at that and go real okay before i would look at neurons and i was assuming i was seeing something that was at least partially true and now i'm realizing it could be like looking at the pixels on my desktop or icons on my desktop and good luck you know going from that to the data structures and then the voltages and i mean good luck there's just no way so what's interesting about this is that our scientific theories are precise enough and rigorous enough to tell us certain limits but and even the limits of the theories themselves but they're not going to tell us what the next move is and that's where scientific creativity comes in so the stuff that i'm saying here for example um is not alien to physicists the physicists are saying precisely the same thing that space time is doomed we've assumed that space time is fundamental we've assumed that for for several centuries and it's been very useful so all the things that you were mentioning the particles and all the work that's been done that's all been done in space time but now physicists are saying space time is doomed there's no such thing as space time fundamentally in the laws of physics and that comes actually out of gravity together with quantum field theory it just comes right out of it it's it's it's a theorem of of those two theories put together but it doesn't tell you what's behind it so the physicists are know that their their best theories einstein's gravity and quantum field theory put together entail that space-time cannot be fundamental and therefore particles in space-time cannot be fundamental they're just irreducible representations of the symmetries of space time that's what they are so we have so space time so we put the two together we put together what the physicists are discovering and we can talk about how they do that and then we the new discoveries from evolution of natural selection both of these discoveries are really in the last 20 years and what both are saying is um spacetime has had a good ride it's been very useful reductionism has been useful but it's over and it's time for us to go beyond when you say spacetime is doomed is it the space is the is the is it the time is it the very hard-coded specification of four dimensions um or are you specifically referring to the kind of uh perceptual domain that humans operate in which is space time you think like there's a 3d um like our world is three dimensional and time progresses forward therefore three dimensions plus one for d what uh what what exactly do you mean by space-time what do you mean by space-time is doomed great great so this is by the way not my quote this is from for example nema or kanye hamed at the institute for advanced study at princeton ed whitten also there david gross nobel prize winner so this is not just something the cognitive scientists this is what the physicists are saying yeah the physicists are space-time uh skeptics we are they're saying that and i can say exactly why they think it's doomed but what they're saying is that you know because your question was what what aspect of space time what are we talking about here it's both space and time they're union into space time as an einstein's theory that's doomed and they're they're basically saying that even quantum theory this is with neymar connie ahmed especially so the hilbert spaces will not be fundamental either so that that the notion of hilbert space which is really critical to quantum field theory quantum information theory uh that's not going to figure in the fundamental new laws of physics so what they're looking for is some new mathematical structures beyond space-time beyond you know einstein's four-dimensional space-time or supersymmetric version geometric algebra signature two comma four kind of there are different ways you can represent it but they're finding new structures and then by the way they're succeeding now they're finding they found something called the amplitude this is nema and his colleagues the the cosmological polytope these are so there are these like polytopes these polyhedra in in multi-dimensions generalizations of simplices that are coding for for example the scattering amplitudes of processes in the large hadron collider and other other colliders so they're finding that if they let go space-time completely they're finding new ways of computing these scattering amplitudes that turn literally billions of terms into one term when you do it in space and time because it's the wrong framework it's it's it's just a user interface from that's now from the evolutionary point of view it's just user interface it's not a deep insight into the nature of reality so it's missing deep symmetry something called a dual conformal symmetry which turns out to be true of the scattering data but you can't see it in space-time and it's making the comp the computations way too complicated because you're trying to compute all the loops and feynman diagrams and all the finement integrals so see the feynman approach to the scattering amplitudes is trying to enforce two critical properties of space time locality and unitarity and so by when you enforce those you get all these loops and multiple you know different levels of loops and for each of those you have to add new terms to your computation but when you do it outside of space time you don't have the notion of unitarity you don't have the notion of locality you have something deeper and is capturing some symmetries that are actually true of the data and but then when you look at the geometry of the facets of these polytopes then certain of them will code for unitarity and locality so it actually comes out of the structure of these deep polytubes so what we're finding is there's this whole new world now beyond space-time that is making explicit symmetries that are true of the data that cannot be seen in space-time and that is turning the computations from billions of terms to one or two or a handful of terms so we're getting insights into symmetries and we're and all of a sudden the math is becoming simple because we're not doing something silly we're not adding up all these loops in space time we're doing something far deeper but they don't know what this world is about also you know they're in an interesting position where we know that space time is doomed and not i should probably tell you why it's doomed what they're saying about why it's doomed but but they need a flashlight to look beyond space-time what what flashlight are we going to use to look into the dark beyond space time because einstein's theory and quantum theory can't tell us what's beyond them all they can do is tell us that when you put us together space time is doomed at 10 to the minus 33 centimeters 10 to the minus 43 seconds beyond that space time doesn't even make sense it just has no operational definition so but it doesn't tell you what's beyond and so they're they're just looking for deep structures like guessing it's really fun so these really brilliant guys generic brilliant men and women who are doing this work physicists are making guesses about these structures informed guesses because they're trying to ask well okay what deeper structure could give us the stuff that we're seeing in space time but without certain commitments that we have to make in space time like locality and so they make these brilliant guesses and of course most of the time you're going to be wrong but once you get one or two that start to pay off and then you get some lucky breaks so they got a lucky break back in 1986 a couple of mathematicians named park and taylor took the scattering amplitude for two gluons coming in at high energy and four gluons going out at low energy so that kind of scattering thing so it's like apparently for people who are into this that's sort of something that happens so often you need to be able to find it and get rid of those because you already know about that and you need to so you needed to compute them it was billions of terms and they couldn't do it even for the supercomputers couldn't do that for the many billions or millions of times per second they need to do it so they they begged the experimentalists begged the theorists please can you you got it and so park and taylor took the billions of terms hundreds of pages and miraculous miraculously turned it into nine and then a little bit later they guessed one term expression that turned out to be equivalent so billions of terms reduced to one term the so-called famous park taylor formula 1986 and that was like okay where did that come from what this is a pointer into a deep realm beyond space and time but but no one i mean what can you do with it and they thought maybe it was a one-off but then other formulas started coming up and then eventually neymar connie hamad and his team found this thing called the amplitude which really sort of captures the whole a big part of the whole balox i'm sure they would say no there's plenty more to do so so i won't say they did it all by any means they're looking at the cosmological polytope as well so what's remarkable to me is that two pillars of modern science quantum field through with gravity on the one hand and evolution by natural selection on the other just in the last 20 years have very clearly said space time has had a good run reductionism has been a fantastic methodology so we had a great ontology of space time a great methodology of reductionism now it's time for a new trick but now you need to go deeper and and show but by the way this doesn't mean we throw away everything we've done not by a long shot every new idea that we come up with beyond space time must project precisely into space time and it better give us back everything that we know and love in space time or generalizations or it's not going to be taken seriously and it shouldn't be so so we have a strong constraint on whatever we're going to do beyond space-time it needs to project into space-time and whatever this deeper theory is it may not itself have evolution by natural selection this may not be part of this deeper realm but when we take the whatever that thing is beyond space-time and projected into space-time it has to look like evolution by natural selection or it's wrong so so that's so that's a strong constraint on this work so even the evolution by natural selection and uh quantum field theory could be interfaces into something that that doesn't look anything like like you mentioned i mean it's interesting to think that evolution might be a very crappy interface into something much deeper that's right they're both telling us that the framework that you've had can only go so far and it has to stop and there's something beyond and that framework the very framework that is space and time itself now of course evolution by natural selection is not telling us about like einstein's relativistic spaces that was another question you asked a little bit earlier it's telling us more about our perceptual space and time which um we have used as the basis for creating first a newtonian space versus time as a mathematical extension of our perceptions and then einstein then took that and extended it even further so the relationship between what evolution is telling us what the physicists are telling us is that in some sense the newton and einstein space time are formulated as sort of rigorous extensions of our perceptual space um making it mathematically rigorous and laying out the symmetries that that they find there so that's sort of the relationship between them so it's the perceptual space time that evolution is telling us is just a user interface effectively and then the physicists are finding that even the mathematical extension of that into the einsteinian formulation has to be as well um not the final story there's something deeper so let me ask you about reductionism and interfaces as we march forward from newtonian physics uh to quantum mechanics these are all in your view interfaces are we getting closer to objective reality how do we know if these interfaces in the process of science the reason we like those interfaces is because they're predictive of some aspects strongly predictive about some aspects of our reality is that completely deviating from our understanding of that reality or is it helping us get closer and closer and closer well of course one critical constraint on all of our theories is that they are empirically tested and past the experiments that we have for them so so no one's arguing against experiments being important and wanting to test all of our our current theories and uh any new theories on that so that's that's that's all there but we have good reason to believe that science will never get a theory of everything you know everything everything everything everything right a final theory of everything right i think that my own take is for what it's worth is that girdles in completeness theorem sort of points us in that direction that even with mathematics uh any finite accidentalization that's sophisticated enough to be able to do arithmetic it's easy to show that there'll be statements that are true that can't be proven can't be deduced from within that framework and if you add the new statements to your axioms then there'll be always new statements that are true but can't be proven with a new axiom system and the best scientific theories indian physics for example and also now evolution are mathematical so our theories are going to be they're going to have their own assumptions and um they'll be mathematically precise and they'll be theories perhaps of everything except those assumptions because the assumptions are we say please grant me these assumptions if you grant me these assumptions then i can explain this other stuff but so you have the assumptions that um [Music] are like miracles as far as the theory is concerned they're not explained they're the starting points for explanation and then you have the mathematical structure of the theory itself which will have the girdle limits and so my my take is that um reality whatever it is is always going to transcend any conceptual theory that we can come up with there's always going to be mystery at the edges right uh contradictions and all that kind of stuff okay and truths so there's an this idea that is brought up in the financial space of uh settlement of transactions it's often talked about in cryptocurrency especially so you could do you know money cash is not connected to anything uh it used to be connected to gold to physical reality but then you can use money to exchange uh to exchange value to transact uh so when it was on the gold standard the money would represent some stable uh component of reality isn't it more effective to avoid things like hyperinflation if we generalize that idea isn't it better to connect your uh whatever we humans are doing in the social interaction space with each other isn't it better from an evolutionary perspective to connect it to some degree to reality so that the the transactions are settled with something that's universal as opposed to us constantly operating in something that's a complete illusion isn't it easy to hyperinflate that like when where you really deviate very very far away from um from the underlying reality or do you not never get in trouble for this can you just completely drift far far away from the underlying reality and never get in trouble that's a great question on the financial side there's two levels at least that we could take your question one one is strictly like evolutionary psychology of financial systems um and that's that's pretty interesting um and there the decentralized idea that the d5 kind of idea in cryptocurrencies may make good sense from just an evolutionary psychology point of view having you know human nature being what it is putting a lot of faith in a few central controllers depends a lot on the veracity of those and trustworthiness of those few central controllers and we have ample evidence time and again that um that's often betrayed so it makes good evolutionary sense i would say to have a decentralized i mean democracy is a step in that direction right we're we don't we don't have a monarch now telling us what to do we decentralize things right because if the monarch if you have marcus aurelius as your emperor you're great if you have nero it's not so great and so we don't want that so democracy is a step in that direction but but i think the defy thing is is an even bigger step and is is going to even make the democratization even even greater so so that's one level also the fact that power corrupts and absolute power corrupts absolutely is also an evolut a consequence of evolution right that's also a feature i think right you can argue from the long span of living organisms it's nice for power to corrupt for you to it so uh mad men and women throughout history might be useful to teach us a lesson we can learn from our negative example right exactly right right right so power does corrupt and i think that you can think about that again from an evolutionary point of view but i think that your question was a little deeper when that was is does the evolutionary interface idea sort of unhinge science from from some kind of important test for the theories right we don't want it doesn't mean that anything goes in scientific theory but there's no if there if we don't see the truth is there no way to tether our theories and test them and and i think there there's no problem there we we can only test things in terms of what we can measure with our senses in space and time so we're going to have to continue to do experiments and but we're going to re we're going to understand a little bit differently what those experiments are we had thought that when we see a pointer on a some machine in a you know an experiment that the machine exists the pointer exists and the values exist even when no one is looking at them and that they're an objective truth and and our best theories are telling us no the pointers pointers are just pointers and that's what you have to rely on for making your judgments um but um that even the pointers themselves are not the objective reality so and and i think girdle is telling us that not that um anything goes but you know as you develop new axiom systems you will find out what goes within that axiom system and what what testable predictions you can make so i don't think we're we're untethered we we continue to do experiments what i think we won't have that we want is a conceptual understanding that gives us a theory of everything that's final and complete i i think that this is to put it another way this is job security for scientists our job will never be done is job security for neuroscience because before we thought that when we looked in the brain we saw neurons and neural networks and and uh you know action potentials and and synapses and so forth and that's that was it that that was the reality now we have to reverse engineer that we have to say what is beyond space time what is going on what is a dynamical system beyond space time that when we project it into einstein space time gives us things that look like neurons and neural networks and synapses that's so we have to reverse engineer so there's going to be lots more work for neuroscience it's going to be far more complicated and difficult and challenging but but that's wonderful that's what we need to do we thought neurons exist when they aren't perceived and they don't in the same way that if i show you when i say they don't exist i should be very very concrete if i draw on a piece of paper a little sketch of something that is called the necker cube it's just a little line drawing of a cube right on a flat piece of paper if i execute it well and i show it to you you'll see a 3d cube and you'll see it flip sometimes you'll see one face in front sometimes you'll see the other face in front but if i ask you know which face is in front when you don't look you know the answer is well neither face is in front because there's no cube this is a flat piece of paper yeah so when you look at the piece of paper you perceptually create the cube and when you look at it then you fix one face to be in front and one face to the other so that's what i mean when i say it doesn't exist space time itself is like the cube it's a data structure that your sensory systems construct whatever your sensory systems mean now because we now have to even not even take that for granted but there are perceptions that you construct on the fly and uh their data structures and the computer sciences and you garbage collect them when you don't need them so you create them and garbage collect them but is it possible that it's mapped well in some concrete predictable way to objective reality the sheet of paper this two-dimensional space or we can talk about space-time maps in some way that we maybe don't yet understand but will one day understand what that mapping is but it maps reliably it is tethered in that way well yes and and so the new theories that the physicists are finding beyond spacetime have that kind of tethering so they're they show precisely how you start with an amplitude and how you project this high dimensional structure into the four dimensions of space time so there's a precise procedure that that relates the two and they're doing the same thing with the cosmological polytopes so so they're the they're the ones that are making the most uh you know concrete and and fun advances going beyond space time and there they're they're tethering it right they say this is precisely the mathematical projection from this deeper structure into space time one thing i'll say about as a non-physicist what that i find interesting is that they're finding just geometry but there's no notion of dynamics right now they're just finding these static geometric structures which is impressive i'm so i'm not putting them down this is what they're doing is unbelievably complicated and brilliant and uh adventurous although it's all those things and beautiful beautiful yeah from a human aesthetic perspective because geometry is beautiful it's it's absolutely and it's f they're finding symmetries that are true of the data that can't be seen in space-time but i'm looking for a theory beyond space time that's a dynamical theory i would love to find and we can talk about that at some point a theory of consciousness in which the dynamics of consciousness itself will give rise to the geometry that the physicists are finding beyond space time if we can do that then we'd have a completely different way of looking at how consciousness is related to what we call the brain or the physical world more generally right right now all of my brilliant colleagues well 99 of them are trying to they're they're assuming space-time is fundamental they're assuming that particles are fundamental quarks gluons leptons and so forth elements atoms and so forth are fundamental and that therefore neurons and brains are part of objective reality and that somehow when you get matter that's complicated enough it will somehow generate conscious experiences by its functional properties or if you're pan psychist maybe you in addition to the physical properties of particles you add your consciousness property as well and then you have you combine these physical and conscious properties to get more complicated ones but they're all doing it within space time all of the work that's being done on consciousness and this relationship to the brain is all assumed something that our fit our best theories are telling us is doomed space time why does that particular assumption bother you the most so you bring up space time i mean that's just one useful interface we've used for a long time uh surely there's other interfaces is space time just the one of the big ones that you to build up people's intuition about the fact that they just do assume a lot of things strongly or or it is is it in fact the fundamental flaw in the way we see the world well everything else that we think we know are things in space time sure and so if you when you say space time is doomed this is a shot to the heart yeah of the whole framework the whole conceptual framework that we've had in science not to the scientific method but to the the fundamental ontology and also the fundamental methodology the ontology of spacetime and its contents and the methodology of reductionism which is that as we go to smaller scales in space-time we will find more and more fundamental laws and it's been very useful for for the space and time for centuries reductionism for centuries but but now we realize that that's over reductionism is in fact dead as is space time what exactly is reductionism what is the process of reductionism that is different than uh some of the physicists that you mentioned that are trying to think trying to let go of the assumption of space-time looking beyond isn't that still trying to come up with a simple model that explains this whole thing isn't it still reducing it's a wonderful question because it really helps to clarify two different notions which is scientific explanation on the one hand and a particular kind of scientific explanation on the other which is the reductionist so the reductionist explanation is saying i will start with with things that are smaller in space-time and therefore more fundamental where the laws are more fundamental so we go uh to just smaller and smaller scales whereas with in science more generally we just say like when einstein did the special theory of relativity he's saying let me have a couple postulates i will assume that the speed of light is universal for for all um [Music] observers in uniform motion um and that the laws of physics so i feel for for uniform motion are are those that's not a reductionist that those are saying grant me these assumptions i can build this entire concept of space time out of it it's not a reductionist thing you're not going to smaller and smaller scales of space you're you're coming up with these deep deep principles same thing with this theory of gravity right it's the falling elevator idea right so this is not a reductionist kind of thing it's it's it's it's it's something different so simplification is a bigger thing than just reductionism reductionism has been a particularly useful kind of scientific explanation for example in thermodynamics right where the notion that we have of heat some macroscopic thing like temperature and heat it turns out that neil boltzmann and others discover well hey you know if we go to smaller and smaller scales we find these things called molecules or atoms and if we think of them as bouncing around having some kind of energy then um what we call heat is is a is really can be reduced to to that and and so that's a particularly useful kind of reduction is a useful kind of scientific explanation that works within a range of scales within space-time but we know now precisely where that has to stop at 10 to the minus 33 centimeters and 10 to the minus 43 seconds and i would be impressed if it was 10 to the minus 33 trillion centimeters i'm not terribly impressed at 10 to the minus 33 centimeters [Laughter] i don't even know how to comprehend either of those numbers frankly uh do just a small aside because i am a computer science person i also find cellular automator beautiful yes and uh so you have somebody like uh stephen wolfram who recently has been very excitedly exploring um a proposal for a data structure that could be um the numbers that would make you a little bit happier in terms of scale because they're very very very very tiny do you like this space of exploration i'm really thinking letting go of space time letting go of everything and trying to think what kind of data structures could be underneath this whole mess that's right so if they're thinking about these as outside of space-time then that's that's what we have to do that's what our best theories are telling us you now have to think outside of space-time now of course i should back up and say we know that einstein surpassed newton right but that doesn't mean that there's not good work to do a newton there's all sorts of newtonian physics that takes us to the moon and so forth and there's lots of good problems that we want to solve with newtonian physics the same thing will be true of space-time we'll we'll still it's not like we're going to stop using space time we'll continue to do all sorts of good work there but for for those scientists who are really looking to go deeper to actually find the next you know just like what einstein did to newton what what are we going to do to einstein how do we get beyond einstein and quantum theory to something deeper then we have to actually let go and and if we're going to do like this a automata kind of approach it's critical that it's not automata in space-time it's automata prior to space-time from which we're going to show how space-time emerges if you're doing automotive within space-time well that might be a fun model but it's not the radical new step that we need yeah so the space time emerges from that whatever system yeah like you're saying it's it's a dynamical system do we even have an understanding what dynamical means when we go beyond when we start to think about dynamics that could mean a lot of things even causality could mean a lot of things if if we if we realize that everything is an interface like what how much do we really know is an interesting question because you brought up neurons i got to ask you and another yet another tangent there's a paper i remember a while ago looking at called uh could a neuroscientist understand a microprocessor and i just enjoyed that thought experiment that they provided which is they basically it's a couple of uh neuroscientists eric jonas and conrad cording uh who used the tools of neuroscience to analyze a microprocessor i saw a computer computer chip now if we lesion it here what happens and so forth and if you go on lesion in the computer it's very very clear that lesion experiments on computers are not going to give you a lot of insight into how it works and also the measurement devices and the kind of sort of just using the basic approaches of neuroscience collecting the data trying to intuit about the underlying function of it and that helps you understand that our scientific exploration of concepts depending on the field uh our are maybe in the very very early stages i wouldn't say it leads us astray perhaps it does sometimes but it's not a uh it's not anywhere close to some fundamental mechanism that actually makes a thing work i don't know if you can sort of comment on that in terms of using neuroscience to understand the human mind and neurons are we really far away potentially from uh understanding in the way we understand the transistors enough to be able to build a computer so one one um one thing about understanding is you can understand for fun the other one is to understand so you could build things and and that's when you really have to understand uh exactly in fact what got me into the field that i at mit was um worked by david marr on this very topic so david maher was a professor at mit but he'd done his phd in neuroscience studying just the architectures of the brain but he realized that his his work it was on the cerebellum um he he realized that his work as as rigorous as it was left him unsatisfied because he didn't know what the cerebellum was for yeah and and and why it had that architecture and so he he went to mit and he was in the ai lab there and and uh he said he had this three-level approach that really grabbed my attention so i when i was an undergrad at ucla i read one of his papers in a class and said who is this guy because he said you have to have a computational theory what is being computed and why an algorithm how is it being computed what what are the percep algorithms and then the hardware how does it get instantiated in the hardware and so to really do he argued we needed to have understanding of all those levels and i that really got me i loved the neuroscience but i realized this guy was saying if you can't build it you don't understand it effectively and so that's why i went to mit and i had the pleasure of working with david until he died as just a year and a half later so there was there's been that idea that you know with neuroscience we we have to have in some sense a top-down model of what what what's being computed and why that we would then go after and same thing with the you know trying to reverse engineer you know a computing system like your laptop we really would we really need to understand what the user interface is about and why we have what are keys on the keyboard for and so forth you need to know why to really understand all the circuitry and what it's what it's for now we don't the evolution of natural selection does not tell us the deeper question that we're asking the answer to the deeper question which is why what what why what what's this deeper reality and what's it up to and why it all it tells us is that whatever reality is it's not what you see what you see is is just an adaptive fiction so just to linger on this fascinating bold question that shakes you out of your dream state does this fiction still help you in building intuitions as literary fiction does about reality the reason we read literary fiction uh is it helps us build intuitions and understanding in indirect ways sneak up to the difficult questions of human nature great fiction same with this observed reality um does this interface that we get this fictional interface help us build intuition about deeper truths of how this whole mess works well i think that each theory that we propose will give its own answer to that question right so when the physicists are proposing um these structures like the amplitude and cosmological polytope sociohedron and so forth beyond space time we can then ask your question for those specific structures and say how much information for example does evolution financial selection and the the kinds of sensory systems that we have right now give us about this deeper reality and and why did we evolve this way we can try to answer that that question from within the deeper so there's not going to be a general answer i think we're going to what we'll have to do is posit these new deeper theories and then try to answer your question within the framework of those deeper theories knowing full well that there will be an even deeper theory so is it is this paralyzing though because how do we know we're not completely adrift uh out to sea lost forever from so like that our theories are completely lost so if if if it's all uh if we can never truly deeply introspect to the bottom if it's always just turtles on top of turtles infinitely um isn't that paralyzing for a scientific mind well it's interesting that you say introspect to the bottom because there there is that there is one i mean again this isn't the same spirit of what i said before which is it depends on what answer you give to what's beyond space time what answer we would give to your question right so but one answer that um is interesting to explore is something that spiritual traditions have said for thousands of years but haven't said precisely so we can't take it seriously in science until it's made precise but we might be able to make it precise and and that is that um they've they've also said something like um space and time aren't fundamental they're maya they're their illusion and but but that um if you look inside if you introspect and let go of all of your particular perceptions uh you will come to something that's beyond conceptual thought and that is they claim uh being in contact with the deep ground of being that that transcends any particular conceptual understanding if that is correct now i'm not saying is correct but and i'm not saying it's not correct i'm just saying if that's correct then it would be the case that as scientists because we also are in touch with this ground of being we would then not be able to conceptually understand ourselves all the way but we could know ourselves just by being ourselves and so we would there would be a sense in which there is a fundamental grounding to the whole enterprise because we're not separate from the enterprise this is the opposite of third the impersonal third person science this this would make science be personal personal all the way down and and but but nevertheless scientific because the scientific method would still be what we would use all the way down for the conceptual understanding unfortunately still don't know if you went all the way down it's possible that this kind of whatever consciousness is and we'll talk about it is getting um the cliche statement of be yourself uh is is it is somehow digging at a deeper truth of reality but you still don't know when you get to the bottom you know a lot of people they'll take psychedelic drugs and they'll say well that takes my mind to certain places where it feels like that is revealing some deeper truth of reality but it's still it could be interfaces on top of interfaces that's that's um in your view of this you really don't know it means gato's and completeness is that you really don't know my own view on it for what it's worth because i don't know the right answer but my own view on it right now is that it um it's never ending i think that there will never that this is great as i said before great um job security for science and that we if this is true and if if consciousness is somehow important or fundamental in the universe this may be an important fundamental fact about consciousness itself that that it's the never-ending exploration that's going on in some sense well well that's interesting you push back on the job security okay so maybe as we understand this kind of idea deeper and deeper we understand that the pursuit is not a fruitful one then maybe we need to maybe that's why we don't see aliens everywhere as you get smarter and smarter and smarter you realize that like exploration is uh there's other fun ways to spend your time than exploring you could be you could be sort of living maximally in some way that's not exploration you know i could there's all kinds of video games you can construct and put yourself inside of them that don't involve you going outside of the game world it's uh you know feeling for my human perspective what seems to be fun is challenging yourself and overcoming those challenges so you can constantly artificially generate challenges for yourself like sisyphus and his boulder just and and that's it so the scientific method that's always reaching out to the stars that's always trying to figure out the puzzle on bottom puzzle the the trick always trying to get to the bottom turtle maybe if we can build more and more the intuition that that's uh infinite pursuit we get um we agreed to start deviating from that pursuit and start enjoying the here and now versus the looking out into the unknown always maybe that's a looking out into the unknown as a early activity for a species that's evolved i'm just sort of saying uh pushing back because you probably got a lot of scientists excited in terms of job security i could i could envision where it's not job security where scientists become more and more useless uh maybe they're like the holders of the ancient wisdom uh that's that allows us to study our own history but not much more than that just to get well let's push back that's good push back i'll i'll put one in there for the scientists again yes but but sure but then i'll take the other side too so when faraday did all of his experiments with magnets and electricity and so forth he came with all this wonderful empirical data and james clerk maxwell looked at it and wrote down a few equations which we can now write down in the single equation the maxwell equation if we use geometric algebra just one equation that opened up unbelievable technologies where you know people are zooming and talking to each other around the world the whole electronics industry there was something that transformed our lives in a very positive way with the theories beyond space time here's one potential right now most of the galaxies that we see um we can see them but we know that we could never get to them no matter how fast we traveled they're going away from us at the speed of light or beyond so we can't we can't ever get to them so there's all this beautiful real estate that's just smiling and waving at us and we can never get to it yeah but that's if we go through spacetime but if we recognize that spacetime is just a data structure it's not fundamental we're not little things inside space time space-time is a little data structure in our perceptions it's just the other way around once we understand that and when we get equations for the stuff that's beyond space time maybe we won't have to go through space time maybe we can go around it maybe i can go to proxima centauri and not go through space i can just go right there directly it's a data structure we can start to play with it so so i think that my for what it's worth my take would be that that the endless sequence of theories that we could contemplate building will lead to an endless sequence of new remarkable insights into the potentialities the possibilities that would that would that would seem miraculous to us and that we will be motivated to continue the exploration partly um just for the technological innovations that that come out but you're the other thing that you mentioned though what about just being what if we when we decide instead of all this doing and exploring what about being my guess is that the best scientists will do both and that the act of being will be a place where they get many of their ideas and that they then pull into the conceptual realm and i think many of the best scientists like einstein comes to mind right where these guys say look i didn't come up with these ideas by a conceptual analysis i was thinking in vague images and i was it was just something non-conceptual and then it took me a long long time to pull it out into concepts and then longer to put it into math but the real insights didn't come from just slavishly right you know playing with equations they came from a deeper place and so there there may be this going back and forth between the complete non-conceptual where there's essentially no end to the wisdom and then conceptual systems where there's the girdle limits um that we have to that and that may be if consciousness is is important than fundamental that may be what consciousness at least part of what consciousness is about is this discovering itself discovering its possibilities so to speak we can talk about what that might mean um by going from the non-conceptual to the conceptual and back back and forth to get better and better and better at being right let me ask you just to linger on the evolutionary because you mentioned evolutionary game theory and that's really where you the perspective from which you come to form the case against reality uh at which point in our evolutionary history do we start to deviate the most from reality is it uh is it way before life even originated on earth is it um in the early development from bacteria and so on or is it when some inklings of what we think of as intelligence or maybe even uh complex consciousness started to emerge so where did this deviation [Music] just like with the interfaces on in a computer you know you start with transistors and then you have assembly and then you have c c plus plus then you have python then you have guise and all that kind you have layers upon layers when do we start to dva well david maher again my advisor at mit in his book vision suggested that the more primitive sensory systems were less realistic less vertical but that by the time you got to something as complicated as the humans we were actually estimating the true shapes and distances to objects and so forth so so his point of view and i think it was probably it's not an uncommon view among my colleagues that that yeah the sensory systems of lower creatures may just not be complicated enough to give them much much truth um but as you get you know to 86 billion neurons you can now compute the truth or at least the parts of the truth that we need when i look at evolutionary game theory one of my graduate students justin mark did some simulations using genetic algorithms so there he was just exploring we start off with random organisms random sensory genetics and random actions and the first generation was unbelievably there were it was a foraging situation they were foraging for resources and most of them you know stayed in one place didn't do anything important and and but we could then just look at how the genes evolved and and what we found was what what what what he found was that uh basically you never even saw the the truth organisms even come on the stage they they if they came but they were gone in one generation they just they just weren't so they they they came and got they came and went uh even just in one generation they just are not good enough the ones that were just tracking their senses just were tracking the fitness payoffs were were far more um fit than than um the truth seekers so from so an answer at one level i want to give an answer at a deeper level but just with evolutionary game theory because my attitude as a scientist is um i don't believe any of our theories i take them very very seriously i study them i look at their implications but none of them are the gospel they're just the latest ideas that we have and you know so the reason i study evolutionary game theory is because that's the best tool we have right now in this area there's there isn't nothing else that competes and so as a scientist it's my responsibility to take the best tools and see what they mean and the same thing the physicists are doing they're taking the best tools and looking at what what they entail but i don't i i think that science now has enough experience to realize that we should not believe our theories in the sense that we've now arrived in 1890 it was a lot of physicists thought we'd arrived they were discouraging um bright young students from going into physics because it was all done and that's precisely the wrong attitude yeah forever this is the wrong attitude forever the attitude we should have is i uh a century from now they'll be looking at us and laughing at what we didn't know and we just have to assume that that's going to be the case uh just just know that everything that we think is so brilliant right now our final theory a century from now they'll look at us like we look at the physicists of 1890 and go how could they have been so dumb yeah so so i don't want to make that mistake so so i'm not doctrinaire about any of our current scientific theories i am dr nir about this we should use the best tools we have right now with with humility well so let me ask you about game theory there's um i love game theory uh evolution game theory um but i'm always suspicious of it um like economics um when you construct models it's too easy to construct things uh that oversimplify just because we our human brains enjoy the simplification of constructing a few variables that somehow represent organisms or represent people and running a simulation that then allows you to build up intuition and it feels really good because you guys can get some really deep and surprising intuitions but how do you know your models aren't the assumptions underlying your models on some fundamentally flawed and because of that your conclusions are fundamentally flawed so i guess my question is what are the limits in your use of game theory evolution game theory your experience with it what are the limits of game theory so i've gotten some pushback from professional colleagues and friends who have tried to rerun simulations and try to i mean the idea that we don't see the truth is not comfortable and so many of my colleagues are very interested in trying to show that we're wrong and so the idea would be to say that somehow we did something as you're suggesting maybe something special that wasn't completely general um we've got some little special part of the whole search space and evolutionary game theory in which this happens to be true but more generally organisms would evolve to see the truth so the the best pushback we've gotten is from a team at yale and uh they suggested that um if you use thousands of payoff functions so we in our simulations we just use a couple one or two because it was the first simulations right so that would be a limit we had one or two payoff functions we showed the result in those at least for the genetic algorithms and they said if you have 20 000 of them then we can find these conditions in which um truth seek seeing organisms would be the ones that that evolved and survived and so we looked at their simulations and and it certainly is the case that you can find special cases in which truth can evolve so when i say it's probability zero it doesn't mean it can't happen it can happen in fact it could happen infinitely often it's just probability of zero so if probability of zero things can happen infinitely often when you say probability of zero you mean probability close to zero to be very precise so for example if i have a unit square on the plane um and i use a measure in which the um on a probability measure in which the area of the region is this probability then if i draw a curve in that unit square it has measure precisely zero precisely not approximately precisely zero and yet it has infinitely many points so there's an object that for that probability measure has probability of zero and yet there's infinitely infinitely many points in it so that's what i what i mean when i say that the things that are probability of zero can happen infinitely often in principle yeah but infinity as far as i and i look outside often i walk around and look at people i haven't never seen infinity in real life that's an interesting issues i've been looking i've been looking ahead i don't notice it infinitely small or the infinitely big and so the tools and mathematics you could sort of apply the same kind of criticism that it is a very convenient interface into our reality that's a big debate in mathematics the intuitionists versus the ones who take for example the real numbers as as as as real and that's that's a fun discussion nicholas giesen has a physicist said really interesting work recently on how if you go with intuitionist mathematics uh you could effectively quantize newton and you find that the newtonian theory and and quantum theory aren't that different once you go with it it's funny it's really quite interesting so so the the issue you raise is a very very deep one and one that i think we should take quite seriously which is you know how should we think about the reality of the contours hierarchy aleph one lf2 and all these all these different infinities versus um just um a more algorithmic approach right so where it's everything's computable in some sense everything's finite as big as you want but but but nevertheless finite so yeah that ultimately boils down to whether the world is discrete or continuous in some general sense and again we can't really know but there's just a mind-breaking thought just common sense reasoning that something can happen and this yet probability of it happening is zero percent that doesn't that doesn't uh compute for common sense computer right um this is where you have to be a sharp mathematician to really and i'm not sharp is one word what i'm saying is common sense computer is i i mean that [Music] in a very kind of in a positive sense because we've been talking about perception systems and interfaces if we were if we are to reason about the world we have to use the best interfaces we got and i'm not exactly sure uh that game theory is the best interface we got for this oh right and applications of mathematics tricks and tools and mathematics the game theory is the best we got when we're thinking about the nature of reality and fitness functions and evolution period right well that's a fair rejoinder and i think that um that was the tool that we used and if if someone says here's a better mathematical tool and here's why this is this mathematical tool better captures the essence of darwin's idea john maynard smith didn't quite get it with evolutionary game theory there's this better this thing now there are tools like evolutionary graph theory which generalize evolutionary game theory and then there's quantum game theory so so you can you can use uh quantum tools like entanglement for example as as a resource in games that that change the very nature of of the solutions of the the optimal solutions of the game theoretic well the the the work from yale is really interesting it's a really interesting challenge of that kind of of these ideas where okay if you have a very large number of fitness functions or let's say you have a nearly infinite number of fitness functions or a growing number of fitness functions what what kind of interesting things uh start to emerging emerging if you are to be an organism if to be an organism that adapts means having to deal with an ensemble of fitness functions right and so we've actually redone some of our own work based on those and this is the the back and forth that we expect in science right and what we found was that they in their simulations they were assuming that you couldn't carve the world up into objects that uh and so we said well let's relax that assumption allow organisms to create data structures that we might call objects and an object would be you take you you would do hierarchical clustering of your fitness payoff functions the ones that have similar shapes if you have 20 000 of them maybe these 50 are all very very similar so i can take all the perception action fitness stuff and make that into a data structure and we'll call that a unit or an object and as soon as we did that then all of their results went away it turned out they were the special case and that the the organisms that were allowed to only see that that were shaped to see only fitness payoffs were the ones that were so so the idea is that objects then what are objects from an evolutionary point of view this bottle we thought that when i saw a bottle it was because i was seeing a true object that existed whether or not it was perceived evolutionary theories suggest a different interpretation i'm seeing a data structure that is encoding a convenient way of looking at various fitness payoffs i can use this for drinking i could use it as a weapon not very good one i could beat someone with head with it um if my goal is mating this is pointless so i'm seeing for what i'm coding here is all sorts of actions and the payoffs that i could get when i pick up an apple now i'm getting a different set of actions and pay and payoffs for when i pick up a rock i'm getting so for every object what i'm getting is a different set of payoff functions um and act with various actions and so once you allow that then what you find is once once again that truth goes extinct and the organisms that just get an interface are the ones that that win but the question uh just sneaking up on this is fascinating from where do fitness functions originate what gives birth to the fitness functions so if there's a giant black box that just keeps giving you fitness functions what are we trying to optimize you said that water uh has different um uses than an apple so there's these objects what are we trying to optimize and why is not reality a really good generator of fitness functions so each theory makes its own assumptions and says grant me this then i'll explain that so evolutionary game theory says grant me fitness payoffs right and grant me strategies with payoffs and i can write down the matrix for this strategy interacts with that strategy these are the payoffs that come up if you grant me that then i can start to to explain a lot of things now you can ask for a deeper question like okay how does physics evolve biology and where do these fitness payoffs come from right now that would be that's a completely different enterprise and of course evolutionary game theory then would be not the right tool for that it would have to be a deeper tool that shows where evolutionary game theory comes from my own take is that there's going to be a problem in doing that because space time isn't fundamental it's just a user interface and that the distinction that we make between living and non-living is not a fundamental distinction it's an artifact of the limits of our interface right so this is a new wrinkle and this is an important wrinkle what it's so nice to take space and time is fundamental because if something looks like it's inanimate it's inanimate and we can just say it's it's not living now it's much more complicated certain things are obviously living i'm talking with you there's i'm obviously interacting with some something that's alive and conscious i think we've let go of the word obviously in this conversation i think nothing is obvious this is obvious that's right but when we get down to you to like you know an ant it's obviously living but i'll say it it appears to be living it won't get down to a virus now people wonder and when we get down to you know protons people say it's not living and and my attitude is look i have a user interface the interface is there to hide certain aspects of reality and others to to to it's an uneven representation put it that way certain things just get completely hidden dark matter and dark energy are most of the energy and matter that's out there our interface just plain flat out hides them the only way we get some hint is because gravitational things are going wrong within our so so most things are outside of our of our interface the distinction between living and non-living is not fundamental it's an artifact of our interface so if so this is the if we really really want to understand where evolution comes from to answer the question the deep question you asked i think the the right way we're going to do that is to come up with a deeper theory than space time in which there may not be the notion of time and show that whatever this dynamics of that deeper theory is and by the way i'll talk about how you could have dynamics without time but the the dynamics of this deeper theory when we project it into in certain ways then we do get space time and we get what appears to be evolution by natural selection so i would love to see evolution by natural selection nature red and tooth and claw people fighting and animals fighting for resources and a whole bit come out of a deeper theory which perhaps it's all cooperation there's no no limited resources and so forth but as a result of projection you get space and time and as a result of projection you get nature red and tooth and claw the appearance of it but it's all an artifact of the interface i like this idea that uh the line between living and non-living is very important because that's that's the thing that would emerge before you have evolution the idea of death so that seems to be an important component of natural selection if and if that emerged because that's also um you know asking the question i guess that i asked where do fitness come fitness functions come from that's like asking the the old meaning of life question right it's a what's the why why why and one of the big underlying lies okay you can start with evolution on earth but without living without life and death without the line between the living and the dead you don't have evolution so what if underneath it there's no such thing as the living and the dead there's no like this concept of an organism period there's a living organism that's defined by a volume in space-time that somehow interacts that over time maintains its integrity somehow it has some kind of history it has a wall of some kind the outside world the environment and then inside there's an organism so you're defining an organism and also you're defining that organism by the fact that it can move and it can be come alive which you kind of think of as moving combined with the fact that it's keeping itself separate from the environment so you can point out that thing is living and then it can also die that seems to be of all very powerful components of space time that enable you to have something like natural selection and evolution well and there's a lot of interesting work some of it um by collaborators of carl fristen and others where they they have um bayes net kind of stuff that they built on in the notion of a markov blanket so you have some states within this network that are inside the blanket then you have the blanket and then the state's outside the blanket and the states inside this markov blanket are conditionally independent of the states outside the blank blanket conditioned on the blanket and what they're what they're looking at is that the dynamics inside of the states inside the markel blanket seem to be trying to estimate properties of the outside and and react to them in a way so it seems like you're doing probabilistic inferences in ways that might be able to to keep your life so there's interesting work going on in in that direction but but but what i'm saying is is something slightly different and that is like when i when i look at you all i see is skin hair and eyes right that's all i see but i know that there's a deeper reality i believe that there's a much deeper reality there's the whole world of your experiences your thoughts your hopes your dreams in some sense the face that i see is is just a symbol that i create right and as soon as i look away that i delete that symbol but i don't delete you i don't delete the conscious experience the the whole world of your so i i'm only deleting an interface symbol but that interface symbol is a portal so to speak not a perfect portal but a genuine portal into your beliefs into your conscious experiences into that's why we can have a conversation we genuinely but your consciousness is genuinely affecting mine and mine is genuinely affecting yours through these icons which which i create on the fly i mean i create your face when i look i delete it i don't create you your consciousness that's there all the time um but but i do so now when i look at a cat i'm creating something that i still call living and i still think it's conscious when i look at an ant i create something that i still would call living but maybe not conscious when i look at something i call a virus now i'm not even sure i would call it living and when i look at a proton i would say i don't even think it's not alive at all it could be that i'm nevertheless interacting with something that's just as conscious as you i'm not saying the proton is conscious the face that i'm creating when i look at you that face is not conscious that face is a data structure in me that face isn't it it's an experience it's not an experiencer similarly a proton is is is something that i create you know when i look or do a collision in the large hadron collider or something like that but what is behind the entity in space time so i've got this space-time interface and i've just got this entity that i call a proton what is the reality behind it well the physicists are finding these big big structures the sociohedron what's behind those could be consciousness what i'm playing with in which case when i'm interacting with a proton i could be interacting with consciousness again to be very very clear because easily i'm not saying a proton is conscious just like i'm not saying your face is conscious your face is a symbol i create and then delete as i look and so your face is not conscious but i know that that face in my interface the lex friedman face that i create is an interface symbol that's a genuine portal into your consciousness the portal is less clear for a cat even less clear for an ant and by the time we get down to a proton the portal is not clear at all but that doesn't mean i'm not interacting with consciousness it just means my interface gave up and there's some some deeper reality that we have to go after so so that so your question really forces out a big part of this whole approach that i'm talking about so it's this portal unconscious i wonder why you can't your portal is not as good to a cat to the cast consciousness than it is to a human does it have to be have to do with the fact that you're human and just similar organisms organisms that similar complexity are able to create portals better to each other or is it just as you get more and more complex you get better and better portals well let me answer one one aspect that i'm more confident about then i'll speculate on that why is why is it that the portal is so bad with protons well and and and elementary particles more generally so quarks leptons and gluons and so forth well the reason for that is because those are just symmetries of space-time more technically they're irreducible representations of the poincare group of space-time so they're just literally representations of the data structure of space-time that we're using so that's why they're not very much insightful they're they're just almost entirely tied to the data structure itself there's there's not much that they're telling you only something about the data structure not behind the data structure it's only when we get to higher levels that we're starting to in some sense build portals to what's behind space time sure yeah so there's more and more um complexity built on top of the interface of space time with the cat so you can actually build a portal right yeah yeah right yeah i this interface of face and hair and so on my skin there is some sinking going on between humans though where we think like you're you're getting a pretty good representation of the ideas in my head and starting to get a foggy view of uh my memories in my head even though you know this is the first time we're talking you start to project your own memories you start to solve like a giant hierarchy of puzzles about a human because we're all uh there's a lot of similarities a lot of it rhymes so you start to make a lot of inferences you build up this model of a person you have a pretty sophisticated model what's going on underneath again i just i wonder if it's possible to construct these models about each other and nevertheless be very distant from an underlying reality there's a lot of work on this so there's some interesting work called signaling games where they they look at how people can coordinate and come to communicate there's some interesting work that was done by some colleagues and friends of mine louis nerens natalia komarova and kimberly jamison where they we're looking at evolving color words so you have a a circle of colors you know this is the color circle and they wanted to see if they could get people to cooperate and how they carved the color circle up into two units of words and so they had a game theoretic kind of thing that they'd had people do and what they found was that when they included so most people are trichromats you have three kinds of cone photoreceptors but there are some a lot of men seven percent of men have are dichromates they might be missing the red cone photoreceptor they found that the dichromats had an outsized influence on the final ways that the whole space of colors was carved up and and labels attached you needed to be able to include the dichromats in the conversation and so they had a bigger influence on how you made the boundaries of the language i thought that was a really interesting kind of insight that there's going to be again a game perhaps a game where evolutionary or genetic algorithm kind of thing that goes on in terms of learning to communicate in ways that that are useful and so yeah you can use game theory to to actually explore that or signaling games um there's a lot of brilliant work on that i'm not doing it but there's work out there so if it's okay let us tackle once more and perhaps several more times after the big topic of consciousness okay this this very beautiful powerful things that perhaps is the thing that makes us human what is it what's the role of consciousness in um let's say even just the thing we've been talking about which is the formation of this interface um but any kind of ways you want to kind of start sure uh tracking and talking about it well let me say first what most of my colleagues say 99 are again assuming that space time is fundamental particles in space-time matter is fundamental and most are reductionist and so the standard approach to consciousness is to figure out what complicated systems of matter with the right functional properties could possibly lead to the emergence of consciousness that's the general idea right so maybe you have to have neurons maybe only if you have neurons but that might not be enough they have to certain kinds of complexity in their their organization and their dynamics certain kind of network abilities for example so there's there are those who say for example that consciousness arises from orchestrated collapse of quantum states of microtubules in neurons certain so this is hamroff and penrose have that's kind of so it's it's a you start with something physical a property of quantum states of neurons of microtubules and neurons and you say that somehow an orchestrated collapse of those is consciousness or conscious experiences or integrated information theory again you start with something physical and if it has the right kind of functional properties that's something they call fee with the right kind of integrated information then you have consciousness or you can be a pan psychist philip gough for example where you you might say well in addition to the particles in space and time those particles are not just matter they also could have say a unit of consciousness and so but once again you're taking space and time and particles as fundamental and you're adding a new property to them see their consciousness and then you have to talk about how when a proton and neutron where proton and electron get together to form hydrogen then how those consciousnesses merge to or interact to create the clutchness of hydrogen and so forth um there's a tension schema theory which again this is how neural network processes representing to the network itself its attentional processes that could be consciousness um there's global workspace theory and neuronal global work experience workspace theory so there's many many theories of this type what's what's common to all of them is they assume that spacetime is fundamental they assume that physical processes and space time is fundamental pan psychism adds consciousness says an additional thing is almost duelist in that regard and my attitude is our best science is telling us that space-time is not fundamental so why is that important here well for centuries deep thinkers thought of earth air fire and water as the fundamental elements it was a reductionist kind of idea nothing was more elemental than those and you could you could sort of build everything up from those when we got the periodic table of elements we realized that um of course we want to study earth air fire and water there's combustion science for fire there's you know um there's sciences for for all these other things water and so forth so we're going to do science with these things but but fundamental no no if you're looking for something fundamental those are the wrong building blocks earth has many many different kinds of elements that project into the one thing that we call earth if you don't understand that there's silicon that there's iron that there's all these different kinds of things that project into what we call earth you're you're you're hopelessly lost you're not fundamental you're not going to get there and then after the periodic table then we came up with quarks leptons and gluons the the particles of the standard you know the standard model of physics and and and so we actually now know that if you really want to get fundamental the periodic table isn't it it's good for chemistry it's just wonderful for chemistry but if you're trying to go deep fundamental what is the fundamental science that's not it you're going to have to go to quarks leptons and gluons and so forth well now we've discovered space-time itself is doomed quarks leptons and gluons are just irreducible representations of the symmetries of space-time so the whole framework on which consciousness research is being based right now is doomed and for me these are my friends and colleagues that are doing this they're brilliant they're absolutely they're they're brilliant i my feeling is i i'm so sad that they're stuck with this old framework because if they they weren't stuck like with earth air fire and water you could actually make progress so it doesn't matter how smart you are if you start with earth air fire and water you're not going to get anywhere right can i actually just uh uh because the word doomed is so interesting let me give you some options multiple choice quiz is space time we could say is reality the way we perceive it doomed um wrong or fake because doomed just means it could still be right and we're now ready to go deeper it would be that so it's not wrong it's not a complete deviation from a journey toward the truth right it's like earth air fire and water is not wrong there is earth air fire and water that's a useful framework but it's not fundamental right well there's also wrong which is they used to believe as i recently learned that george washington was the president of the first president in the united states was bled to death uh for something that could have been easily treated uh because it was believed that you can get actually i need to look into this further but i guess you get toxins out or demons out i don't know what you're getting out with the bleeding of a person right but so that end up being wrong but widely believed as a medical tool so it's also possible that our assumption of space-time is not just doomed but it's wrong well if we believe that it's fundamental that's wrong but if we believe it's a useful tool that's right but he could see but bleeding somebody to death was believed to be a useful tool and that was true it wasn't just not fundamental right it was very i'm sure there's cases in which bleeding somebody would work but it would be a very tiny tiny tiny percentage of cases so it could be that it's wrong like it's a side road that's ultimately leading to a dead end as opposed to a truck stop or something that you can get off of my feeling is not the dead end kind of thing i i think that the what the physicists are finding is that there are these structures beyond space time but they project back into space time and so space time when they when they say space time is doomed they're explicit they're saying it's doomed in the sense that we thought it was fundamental it's not fundamental right it's a useful absolutely useful and brilliant data structure but there are deeper data structures like cosmological polytube and and and space-time is not fundamental what is doomed in the sense that it's wrong is reductionism which is saying space-time is fundamental right right the idea that that somehow being smaller in space and time or space time is a fundamental nature of reality that's is a that's that's just wrong it turned out to be a useful heuristic for for thermodynamics and so forth and several other places it reductionism has been very useful but that's in some sense an artifact of how we use our interface um yeah so you're saying size doesn't matter okay this is very important for me ultimately ultimately right i mean it's useful for for theories like thermodynamics and also for understanding brain networks in terms of of individual neurons and neurons in terms of chemical systems inside cells that's all very very useful but but the idea that we're getting to the more fundamental nature of reality no when you when you get all the way down in that direction you get down to the quarks and gluons what you realize is what you've gotten down to is not fundamental reality just the irreducible representations of a data structure that's all you've gotten down to so you're always stuck inside the data structure so you seem to be getting closer and closer i went from neural networks to neurons neurons to chemistry chemistry to particles particles to clerks and gluons i'm getting closer and closer to the ground no i'm getting closer and closer to the the actual structure of the data structure of space and time the irreducible representation so that's what you're getting closer to not to a deeper understanding of what's beyond space-time we'll also refer we'll return again to this question of dynamics because you keep saying that space-time is doomed but mostly focusing on the space part of that it's very interesting to see why why time gets the bad cred too because how do you have dynamics without time is the thing i'd love to talk to you a little bit about but let us return um your brilliant whirlwind overview of the different theories of consciousness that are out there what is consciousness if outside of space-time if we think that we want to have a model of consciousness we as scientists then have to say what do we want to write down what what what kind of mathematical model are we going to write down right and if you think about it there's lots of things that you might want to write down about consciousness really complicated subject so most of my colleagues are saying let's start with matter or neurons and and say what properties of matter could create consciousness but i'm saying that that whole thing is out space time is doomed that whole thing is out we need to look at consciousness qua consciousness in other words not as something that arises in space and time but perhaps it's something that creates space and time as a data structure so what do we want and here again there's no hard and fast rule but what you as a scientist have to do is to pick what you think are the minimal assumptions that are going to allow you to boot up a comprehensive theory that is the trick so what do i want so what what i chose to do was to have three things i said that there are conscious experiences feeling of headache the smell of garlic um experiencing the color red there are those are conscious so that's a primitive theory and the reason i want few primitives why because those are the miracles of the theory right the primitives the assumptions of your theory are the things you're not going to explain those are the things you assume and those experiences you particularly mean there's a subjectiveness to them that's what's the thing when people refer to the heart problem of consciousness is it feels like something to look at the color red okay exactly right it feels like something to have a headache or to feel upset to your stomach it feels like something and so though so i'm going to grant that in this theory there are experiences and they're fundamental in some sense so conscious experience so they're not derived from physics they're not functional properties of particles they are sewed generous they're they exist just like we assume space-time exists i'm now saying space-time is just a data structure it doesn't exist independent of conscious experiences sorry to interrupt once again but should we be focusing in your thinking on humans alone or is there something about in relation to other kinds of organisms that have a sufficiently high level of complexity or even or is there some kind of uh generalization of the pan cyclist idea that all consciousness permeates all matter outside of the usual definition of what matter is inside space-time so it's beyond human consciousness human consciousness from my point of view would be one of a countless variety of consciousnesses and even within human consciousness there's a there's countless variety of consciousness within us right you have your left and right hemisphere and apparently if you split the corpus callosum the the personality of the left hemisphere and the religious beliefs of the left hemisphere can be very different from the right hemisphere and their conscious experiences can be disjoint one could have one conscious experience they can play 20 questions the left hemisphere can have an idea in its mind and the right hemisphere has to guess and it might not get it so so even within you there is more than just one consciousness it's lots of consciousnesses so i the the general theory of consciousness that i'm after is not just human consciousness it's going to be just consciousness and i presume human consciousness is a tiny drop in the bucket of the infinite variety of consciousnesses that said i should clarify that the black hole of consciousness is uh the the home cat i'm pretty sure cats lack uh is the embodiment of evil and lack all capacity for consciousness or uh compassion so i just want to lay that on there but that's the theory i'm working i i don't have any good evidence that's just a shout out sorry to distract so that's the first assumption first assumption that's right the second assumption is that these experiences have consequences so i'm going to say that conscious experiences can trigger other conscious experiences somehow so so really in some sense there there's two basic assumptions there's some kind of causality is there's a chain of causality does this relate to dynamics i'll say there's a probabilistic relationship okay um and then so i'm trying to be as non-specific to begin with and see where it leads me so what i can write down are probability spaces so probability space which contains the conscious experiences that this consciousness can have so i i'll i call this a conscious agent this technical thing now i annika harris and i've talked about this and and she rightly cautions me that people will think that i'm bringing in a notion of a cell for agency and so forth when i say conscious agent so i just want to say that i use the term conscious agent merely as a technical term there is no notion of self in my fundamental definition of a conscious age there are only experiences and probabilistic relationships that of how they trigger other experiences so the agent is the generator of the conscious experience the agent is a mathematical structure that includes a probability measure a probability space of possible conscious experiences and what in a markovian kernel which describes how if this agent has certain conscious experiences how that will affect the experiences of other conscious agents okay including itself but you don't think of that as a self no there there is no notion of of a self here there's no notion of of really of an agent but is there a locality is there an organization so this is this is um these are conscious units conscious entities but they're distinct in some way because they have to interact well so here's the interesting thing when we write down the mathematics um when you have two of these conscious agents interacting they s the the pair satisfy the definition of a conscious agent so they are a single conscious agent so there is one conscious agent yeah but it it has a nice analytic decomposition into as many conscious agents that's a nice interface it's a very useful scientific interface yeah it's it's a scale-free or if you like a fractal-like approach to it in which we can use the same unit of analysis at all scales in in studying consciousness but if i want to talk about so there's no notion of learning memory problem solving intelligence self agency so none of that is fundamental so and the reason i did that was because i want to assume as little as possible everything i assume is a miracle in the theory it's not something you explain is something you assume so i have to build networks of conscious agents if i want to have a notion of a self i have to build a self i have to build learning memory problem solving intelligence and planning all these different things i have to build networks of conscious agents to do that it's a trivial theorem that networks of conscious agents are computationally universal that's trivial so anything that we can do with neural networks or you know automata you can do with networks of conscious agents that's trivial but but you can also do more the events in the probability space need not be computable so the markovian dynamics is not restricted to computable functions because the very events themselves need not be computable so so this can capture any computable theory anything we can do with neural networks we can do with conscious agent networks but but it leaves open the door for the possibility of non-computable interactions between conscious agents so we have to to if we want a theory of of memory we have to build it and there's lots of different ways you could build we've actually got a paper chris fields took the lead on this and he we have a paper called conscious agent networks where where chris takes the lead and shows how to use these networks of conscious agents to build memory and to build primitive kinds of of learning but can you provide some intuition of what conscious networks network of conscious networks of conscious agents helps you well first of all what that looks like and i don't just mean mathematically of course maybe that might help build up intuition but how that helps us potentially solve the heart problem of consciousness right or is that baked in that that exists is the can you solve the hard problem of consciousness why it tastes delicious when you eat a delicious ice cream with networks of conscious agents or is that taken as an assumption so the the standard way the heart problem is thought of is we're assuming space and time and particles or neurons for example these are just physical things that that have no consciousness and we have to explain how the conscious experience of the taste of chocolate could emerge from those so that's the typical hard problem of consciousness is that problem right how do you boot up the taste of chocolate the experience of the taste of chocolate from neurons say or the right kind of artificial intelligence circuitry how do you boot that up that's that's typically what the hard problem of consciousness means to to researchers notice that i'm changing the problem i'm not trying to boot up conscious experiences from the dynamics of neurons or silicon or something like that i'm saying that that's the wrong problem my heart problem would go in the other direction if i start with conscious experiences how do i build up space and time how do i build up what i call the physical world how do i build up what we call brains because i'm saying consciousness is not something that brains do brains are something that consciousness makes up it's one among the experience it's an experience an ephemeral experience in consciousness i look inside so i'll to be very very clear right now i have no neurons if you looked you would see neurons that's a data structure that you would create on the fly and it's a very useful one as soon as you look away you garbage collect that data structure just like that necro cube that i was talking about on the piece of paper when you look you see a 3d cube you create it on the fly as soon as you look away that's gone when you say you you mean a human being scientist right now that's right more generally it'll be conscious agents because as you as you pointed out in my asking for a theory of conscience only about humans no it's it's consciousness which human consciousness is just a tiny sliver so but you are saying that there is that's a useful data structure how many other data structures are there that's why i said you human if there's another earth if there's another alien civilization and doing these kinds of investigations would they come up with similar data structures probably not what is the space of data structures i guess is what i'm asking um my my guess is that if consciousness is fundamental consciousness is all there is then the only thing that mathematical structure can be about is possibilities of consciousness and that suggests to me that there could be an infinite variety of consciousnesses and a vanishingly small fraction of them use space-time data structures and the kinds of structures that we use there's an infinite variety of data structures now this is very similar to something that max tegmark has said but i want to distinguish it he has this his level 4 multiverse idea he thinks that mathematics is fundamental and and so that's the fundamental reality and since there's an infinite variety of endless variety of mathematical structures there's an infinite variety of multiverses in his view i'm saying something similar in spirit but importantly different there's an infinite variety of mathematical structures absolutely but mathematics isn't the fundamental reality in this in this framework consciousness is and mathematics is to consciousness like bones are to an organism you need the bones so mathematics is is not divorced from consciousness but it's not the entirety of consciousness by any means and so there's an infinite variety of of conscious consciousnesses and and signaling games that consciousnesses could interact via and therefore worlds you know common worlds data structures that they can use to to communicate so space and time is just one of an infinite variety and so i i think that what we'll what we'll find is that as we go outside of our little space-time bubble um we will encounter utterly alien forms of conscious experience that we may not be able to really comprehend in the in the following sense if i if i ask you to imagine a color that you've never seen before does anything happen right nothing happens nothing happens and and that's just one color i'm asking for just a color we we actually know by the way that there apparently there are women um called tetraphams um who have four color receptors not just three and kimberly jamison and others who studied these women have good evidence that they apparently have a new dimension of color experience that the rest of us don't have so so these women are apparently living in a world of color that you and i can't even concretely imagine no man can imagine them yeah and and yet they're real color experiences and so in that sense i'm saying now take that little baby step oh there are women who have color experiences that i could never have well that's shocking now take that infinite there are consciousnesses where every aspect of their their experiences or is like that new color it's something utterly alien to you you you'd have nothing like that and yet these are all possible varieties of conscious experience when you say there's a lot of consciousnesses it's a singular consciousness basically the set of possible experiences you can have in a subjective way as opposed to the met the underlying mechanism because you say that you know having uh extra color receptor uh ex ability to have new experiences that's somehow a different consciousness is there a way to see that as all the same consciousness the subjectivity itself right because when we have two of these conscious agents interacting in the mathematics they actually satisfy the definition of a conscious agent so in fact they are a single conscious agent so so in fact one way to think about what i'm saying i'm postulating with my colleagues jayton and chris and others robert prentner and so forth there is one big conscious agent infinitely complicated but fortunately we can for analytic purposes break it down all the way to in some sense the simplest conscious agent which has one conscious experience one i like this this one agent can experience red 35 that's it that's what that's what it experiences you can get all the way down to that so you think it's possible that consciousness whatever that is is uh much more is fundamental or at least much more in the direction of the fundamental than his space time as we perceive it that's the proposal and therefore what i have to do in terms of the heart problem of consciousness is to show how dynamical systems of conscious agents could lead to what we call space and time and neurons and brain activity in other words we have to show how you get space-time and physical objects in entirely from a theory of conscious agents outside of space time with the dynamics outside of space-time so that's that's and i can tell you how we plan to to do that but but that's that's the idea okay the magic of it that chocolate is delicious so so there's a mathematical kind of thing that we could say here how it can emerge within the system of networks of conscience agents but um is there going to be at the end of of the proof why chocolate is so delicious or or no i guess i'm going to ask different kinds of dumb questions to try to sneak up oh well that's the right question and when i say that i took conscious experiences as fundamental what that means is in the current version of my theory i'm not explaining conscious experiences where they came from that's the miracle that's one of the miracles so i have two miracles in my theory there are conscious experiences like the taste of chocolate and that the there's a probabilistic relationship when certain conscious experiences occur others are more likely to occur those are the two miracles that are possible to get beyond that and somehow start to chip away at the miracleness of that miracle that chocolate is still delicious i hope so i've got my hands full with what i'm doing right now but but the i can just say at top level how i would think about that that would get at this consciousness without form this is going to be really this is really tough because it's consciousness without form versus the various forms that consciousness takes for the experiences that it has right right so there's so when i write down a probability space for these conscious experience i say here's a probability space for the possible conscious experiences right it's just like when i write down a probability space for an experiment like i'm going to flip a coin twice right and i want to look at the probabilities of various outcomes so i have to write down the probability space there could be heads heads heads tails tails heads tails tails so you before as any class and probability you're told write down your probability space if you don't write down your probability space you can't get started so here's my probability space for consciousness how do i want to interpret that structure the structure is just sitting there there's going to be a dynamics that happens on it right experiences appear and then it disappear just like heads appears and disappears so so one way to think about that fundamental probability space is that corresponds to consciousness without any content the infinite consciousness that transcends any particular content well do you think of that as a mechanism as a thing like the the rules that govern the dynamics of the thing outside of space time isn't that if you think consciousness is fundamental isn't it essentially getting like it is solving the hard problem which is like from uh where does this thing pop up which is the mechanism of the thing popping up whatever the consciousness is the different kinds it was so on that mechanism and also the the question i want to ask is uh how how tricky do you think it is to solve that problem you solved a lot of difficult problems throughout the history of humanity there's probably more problems to solve left than we've solved by like an infinity uh but along that long journey of intelligent species how when when we solve this consciousness one which is one way to measure the difficulty of the problem so i'll give two answers there's one problem i think we can solve but we haven't solved yet and that is the reverse of what my colleagues call the heart problem the problem of how do you start with conscious experiences in the way that i've just described them in the dynamics and build up space and time and brains that i think is a tough technical problem but some principle solvable so i think we can solve that so we would solve the hard problem not by showing how brains create consciousness but how networks of conscious agents create what we call the the symbols that we call brains so that that i think but that does that allow you to so that's interesting that's an interesting idea consciousness creates the brain not the brain creates consciousness but does that allow you to build the thing my guess is that it will enable unbelievable technologies once and i'll tell you why i think it plugs into the work that the physicists are doing so this theory of consciousness will be even deeper than the structures that the physicists are finding like the amplitude but the other but the other answer to your question is less positive i as i said earlier i think that there is no such thing as a theory of everything so that i think that my the theory that my team is working on this conscious station theory is just a 1.0 theory theory we're using probability spaces and markovian curls i can easily see people now saying well we can do better if we go to category theory and we can get a deeper perhaps more interesting and then someone will say well now i'll go to topoi theory and then they'll be so i imagine that there'll be you know conscious agents five ten 3 trillion 0.0 but i think it will never end i think ultimately this question that we sort of put our fingers on of how does the formless give birth to form to the taste the wonderful taste of chocolate i think that we will always go deeper and deeper but we will never solve that that that in some sense that will be a primitive i hope i'm wrong maybe i'm maybe it's just the the limits of my current imagination um so i'll just say my imagination right now doesn't appear that deep hopefully so i don't by the way i'm saying this i don't want to discourage some brilliant 20 year old who then later on proves me dead wrong i hope to be proven dead right just like you said essentially from now everything we're saying now everything you're saying all your theories will be laughing stock they will respect the uh the the the puzzle solving abilities and how how much we were able to do with so little but uh outside of that it will all be just uh the silliness will be entertainment for a teenager especially the silliness when we thought that we were so smart and we knew it all so it would be interesting to explore your ideas by contrasting you mentioned annika monica harris you mentioned um phil goff so outside of if you're not allowed to say the fundamental disagreement is the fact that space time is fundamental um what are interesting distinctions between ideas of consciousness between you and hanukkah for example you guys have uh you've been on a podcast together i'm sure in in in in private you guys have some incredible conversations so what where are some sticky interesting sticking points some interesting disagreements let's say with annika first maybe there'll be a few other people well annika and i just had a conversation this morning where we were talking about our ideas and what we discovered really in our conversation was that um we're pretty much on the same page it was really just consciousness unconsciousness yeah we're our ideas about consciousness are pretty much on the same page she rightly has cautioned me to when i talk about conscious agents to point out that the notion of agency is not for is not fundamental in my my theory the notion of self is not fundamental that's that's absolutely true i can use this network of conscious agents this and i now use tech as a technical term conscious agent is a technical term for that probability space with the markovian dynamics i can use that to build models of a self and to build models of agency but they're not they're not fundamental so she she has really um more been very helpful in in helping me to be a little bit clear about these ideas and not say things that are misleading sure the word i mean this is the interesting thing about language actually is that language quite obviously is an interface yes um to truth it's it's so fascinating that individual words can have so much ambiguity and the the and the slight the specific choices of a word within a particular sentence within the context of a sentence can have so much uh such a difference in meaning it's quite fascinating especially when you're talking about topics like consciousness because it's a very loaded term it means a lot of things to a lot of people and the entire concept is shrouded or mystery so combination of the fact that it's a loaded term and that there's a lot of mystery people can just interpret it in all kinds of ways and so you have to be both precise and help them avoid getting um stuck on some kind of side road of miscommunication lost in translation because you used the wrong word that's interesting i mean because for for a lot of people consciousness is ultimately connected to a self i mean that's our our experience of consciousness is very it's connected to this ego i mean i just i mean what else could it possibly be i can't even how do you begin to comprehend to visualize to conceptualize a consciousness that's not connected to like this particular organism well i have a way of thinking about this whole problem now that's that's that comes out of this this framework that's different so we can imagine a dynamics of consciousness not in space and time just abstractly it could be cooperative for all we know it could be very friendly i don't know but it and you can set up a dynamics a markovian dynamics that is so called stationary and that's a technical term which means that the entropy effectively is not increasing there is some entropy but it's constant so there's no increasing entropy and in that sense the dynamics is timeless there is no entropic time but it's a trivial theorem three line proof that if you have a stationary markovian dynamics any projection that you make of that dynamics by conditional probability and if you want i can state a little bit more even more mathematically precisely for for some readers for listeners but if any projection you take by conditional probability the induced image of that markov chain will have increasing entropy you will have entropic time so so i'll be very very precise i have a mark of chain x1 x2 through xn where x and n goes to infinity right [Music] the entropy h capital h of xn is equal to the entropy h of xn minus 1 for all n so the entropy is the same but it's it's a theorem that h of x n say given x sub 1 is greater than or equal to h of x n minus 1 given x 1. sure where does the greater come from because with the theorem of three-line proof h of x n x1 is greater than or equal to h of xn given x1 and x2 because conditioning reduces but then h of xn minus 1 given x1 comma x2 is equal to h of xn um given x2 xn minus 1 given x2 by the markov property and then because it's stationary it's equal to uh h of x um um i have to write it down next time sure right anyway there's a three-line proof sure so but the assumption of stationarity we're using a lot of times that people will understand right doesn't matter the uh so there's some kind of some markovian dynamics is uh basically trying to model some kind of system with some probabilities and there's agents and they interact in some kind of way and you can say something about that system as it involves uh stationarity so a stationary uh system is one that has certain properties in terms of entropy very well but we don't know if it's stationary now we don't know what properties uh right uh you have to kind of take assumptions and see okay well what is what does the systems behave like under these different properties the the more constraints the more assumptions you take the more predictive the more interesting powerful things you can say but sometimes the limiting that said we're talking about consciousness here how does that you said cooperative okay competitive it just i like chocolate i'm sitting here i have a brain i'm wearing a suit it sure as hell feels like i'm myself now what am i tuning in am i plugging into something am i a projection a simple trivial projection into space-time from some much larger organism that i can't possibly comprehend how the hell you're saying some yes you're building up mathematical intuitions fine great but i'm just i'm i'm having an existential crisis here and i'm going to die soon we'll all die pretty quickly so i i want to i want to figure out why chocolate is so delicious uh so help me out here so let's just keep sneaking up to this right so the whole technical technical thing was to say this even if the dynamics of consciousness is stationary so that there is no entropic time any projection of it any view of it will have the artifact of entropic time that's a limited resource limited resources so that the fundamental dynamics may have no limits limited resources whatsoever any projection will have certainly time is a limited resource and probably a lot of other limited resources hence we could get competition and evolution and nature red and tooth and claw as an artifact of a deeper system in which those aren't fundamental and and in fact i take it as something that this theory must do at some point is to show how networks of conscious agents even if they're not resource limited give rise to evolution by natural selection via a projection yeah but you're saying i'm trying to understand how the limited resources that give rise to um so first the thing gives rise to time that gives rise to limited resource that gives rise to evolution by natural selection how that has to do with the fact that chocolate is delicious well well it's it's not going to do that directly it's going to get to this notion of self so oh oh it's going to give you the notion of so evolution gives you the notion of self and also of of of a self separate from other selves so so the idea would be that competition has life and death all those kinds of things that's right so it it won't i i don't think as i said i don't think that i can tell you how the formless gives rise to the experience of chocolate right now my current theory says that's that's one of the miracles i'm assuming yeah that's so my theory can't do it and the reason my theory can't do it is because hoffman's brain can't do it right now [Laughter] but but the notion of self yes the notion of self can be an artifact of the projection of a so there's one conscious agent because any time conscious agents interact they form a new conscious agent so there's one conscious agent any projection of that one conscious agent gives rise to time even if there wasn't any time in that one conscious agent and it gives rise i i want to now i haven't proven this so this is so now this is me guessing where the theory is going to go not i haven't done this there's no paper on this yet so now i'm speculating my guess is i'll be able to show or the or the my brighter colleagues working with me will be able to show that that we will get evolution of natural selection the notion of individual selves individual physical objects and so forth coming out as a projection of this thing and and that the self this then will be be really interesting in terms of how it starts to interact with certain um spiritual traditions right where they will say that there is a notion of self that needs to be let go which is this finite self that's competing with other selves to you know get more money and prestige and so forth that self in some sense has to die but there's a deeper self which is the timeless um being that preclude that precedes not precludes but precedes um any particular conscious experiences the ground of all experience that there's that notion of a deep capital self but our little capital lowercase s selves could be artifacts of projection and it may be that what consciousness is doing in this framework is right it's it's projected itself down into a self that calls itself dawn and a self that calls itself lacks and through conversations like this it's trying to find out about itself and eventually transcend the limits of the don and lex little icons that it's using and and that little projection of itself through this kind of conversation somehow it it's learning about itself so that that thing dressed me up today in order to understand itself and in some sense you and i are not separate from that thing and we're not separate from each other yeah well i have to question the fashion choices on my end then all right so uh you mentioned you agree on in terms of consciousness on a lot of things with anika is there somebody friend or friendly foe that you disagree with in some nuanced interesting way or some major way about consciousness about these topics of of reality that you return to um often it's like uh uh christopher hitchens with the rabbi david walby have had interesting conversations through years that that added to the complexity and the beauty of their friendship is there is there somebody like that um that over the years has been a source of disagreement with you that strengthened your ideas my ideas have been really shaped by several things one is um [Music] the physicalist framework that my scientific colleagues almost to a person have adopted and that i adopted to until i the reason i walked away from it was because i uh it became clear that we couldn't start with unconscious ingredients and boot up consciousness can you define physicalists in contrast to reductionist so a physicalist i i would say is someone who takes space time and the objects within space time as ontologically fundamental right and then reductionist is saying the smaller the more fundamental that's a methodological thing that's that's saying within space-time as you go to smaller and smaller scales in space you get deeper and deeper laws more and more fundamental laws and you know the reduction of temperature to particle movement was an example of that but i think that that that the reason that worked was almost an artifact of the nature of our interface that was for a long time many of your colleagues including yourself or physicalist and now you broke away broke away because i think you can't start with unconscious ingredients and boot up consciousness and so even with roger penrose where there's like a gray area right and here's the the challenge i would put to to all of my friends um and colleagues who are give one specific conscious experience that you can boot up right so if you think that it's integrated information and this i've asked this of julia to know me a couple times back in the 90s and then just a couple years ago i asked julio okay so great and great information so we're all interested in explaining some specific conscious experiences so what what is you know pick one the taste of chocolate what is the integrated information [Music] precise structure that we need for chocolate and why does that structure have to be for chocolate and why why is it that it could not possibly be vanilla is there any elastomer is there any one specific conscious experience that you can account for because notice they've set themselves the task of booting up conscious experiences from physical systems that's the test they've set themselves but that doesn't mean they're uh i understand your intuition but that doesn't mean they're wrong just because they can't find a way to boot it up yet that's right no that doesn't mean that they're wrong it just just means that um they haven't done it i think it's principled the reason is principled but but i'm happy that they're exploring it but the fact is the remarkable fact is there's not one theory so integrated information theory collapse of microtubules um global workspace theory these are all theories of consciousness these are all theories of consciousness there's not a single theory that can give you a specific conscious experience that they say here is the physical dynamics or the physical structure that must be the taste of chocolate or whatever one they want so you're saying it's impossible they're saying it's just hard yeah my attitude is okay no one said you had to start with neurons or physical systems and boot up consciousness you guys are just you chose that problem so since you chose that problem how much progress have you made well when you've not been able to come up with a single specific conscious experience and you've had these brilliant people working on it for decades now that's not really good progress let me ask you to be to play devil's advocate can you try to steal man steel man meaning argue the best possible case for reality the opposite of your book title so um or maybe just sticking to consciousness can you take the physicalist view can you steal man the physicalist view for a brief moment playing devil's advocate to or um steel man the person you used to be right the physicalist what's the good like saying that you might be wrong right now what would be a convincing argument for that well i think the the argument i would give and that i believed was look when you have very simple physical systems like a piece of dirt there's not much evidence of life for consciousness it's only when you get really complicated physical systems like that have brains and really the more complicated the brains the more it looks like there's consciousness and the more complicated that consciousness is surely that means that simple physical systems don't create much consciousness or if maybe not any or maybe pan cyclists they create the most elementary kinds of simple conscious experiences but you need more complicated physical systems to boot up to create more complicated consciousnesses i think that's the intuition that drives most of my colleagues and you're saying that this concept of complexity is ill-defined when you ground it to space-time well well i think it's well-defined within the framework of space-time right no it's ill-defined relative to what you need to actually understand consciousness because you're grounding complexity in just in space-time well got you right right yeah yeah what i'm saying is if it were true that space-time was fundamental then i would have to agree that if there is such a thing as consciousness given the data that we've got that you know complex brains have consciousness and you know dirt doesn't that somehow is the complexity of the dynamics or organization the function of the physical system that somehow is creating the consciousness um so under those assumptions yes but when the physicists themselves are telling us that space time is not fundamental then i can understand see then the whole picture starts to come into focus why my colleagues are brilliant right these are really smart people i mean francis crick worked on this for the last 20 years of his life these are not stupid people these are brilliant brilliant people the fact that we've come up with not a single specific conscious experience that we can explain and no hope there's there's no one that says i'm really close so i'll have it for you in a year no there's just like there's this fundamental gap so much so that steve pinker in in one of his writings says look he likes the global workspace theory but he says the last dollop of the theory in which you know there's something it's like to he said we may have to just stipulate that as a as a brute fact i mean he that's i mean that one and pinker is brilliant right he he's he understands the state of play on this problem of the hard problem of consciousness starting with physicalist assumptions and then trying to boot up consciousness and you've set yourself the problem i'm starting with physical stuff that's not that's not conscious i'm trying to get the taste of chocolate out as maybe some kind of function of that of the dynamics of that we've not been able to do that and so pinker is saying we may have to punt we may have to just stipulate that last bit he calls it the last dollop um and just say stipulate it as a bare fact of nature that there is something it's like well from my point of view as the physical the whole point the whole promise of the physicalist was we wouldn't have to stipulate i was going to start with the physical stuff and explain where the consciousness came from if i'm going to stipulate consciousness why don't i just stipulate consciousness and not stipulate all the physical stuff too so i'm stipulating less i'm saying okay i agree perspective well it's it's actually what i call the conscious realist perspective consciousness pan cyclists are effectively duelists right they're saying there's physical stuff that really is fundamental and then consciousness stuff so i would go with pinker and say look let's just stipulate the consciousness stuff but i'm not going to stipulate the physical stuff i'm going to actually now show how to boot up the physical stuff from just the consciousness stuff so i'll stipulate less is it possible so if you stipulate less is it possible for our limited brains to visualize reality as we delve deeper and deeper and deeper is it possible to visualize somehow with the tools of math with the tools of computers with the tools of our mind are we hopelessly lost you said there's ways to intuit what's true using mathematics and probability and um sort of uh markovian dynamics all that kind of stuff but that's not visualizing that's what's the kind of building intuition but is it possible to visualize in the way we visualize so nicely in in space time in four dimensions in two in three dimensions sorry well two we really are looking through a two-dimensional screen until what we intuit to be a three-dimensional world and and also inferring dynamic stuff making it 4d anyway is it possible to visualize some pretty pictures that give us a deeper sense of the truth of reality i think that we will incrementally be able to do that i think that for example the picture that we have of electrons and photons interacting and scattering wasn't it may have not been possible until faraday did all of his experiments and then maxwell wrote down his equations and and we were then sort of forced by his equations to think in a new way and then then when planck in 1900 you know desperate to try to solve the the problem of black body radiation that what they call the ultraviolet catastrophe where newton was predicting infinite energies where there weren't infinite energies in black body radiation and he in desperation proposed packets of energy the then once once you've done that and then you have an einstein come along five years later and show how that explains the photoelectro photoelectric effect and then then eventually in 1926 you get quantum theory and then you get this whole new way of thinking that was you know from the newtonian point of view completely contradictory and and were counter-intuitive certainly yeah and maybe if jesus is right not contradictory maybe if you use intuitionist math they're not contradictory but still certainly you wouldn't have gone there and so here's a case where the experiments and then a desperate mathematical move sort of we use those as a flashlight into the deep fog right where and and so that science may be um the flashlight into the deep fog yeah i wonder if it's still possible to visualize in the in the deli like uh we talk about consciousness in from a self perspective experience that hold that idea in our mind the way you can experience things directly we've evolved to experience things in this 3d world and it's that's a very rich experience when you're thinking mathematically [Music] you still in the end of the day have to project it down to a low dimensional space to make to make conclusions their conclusions will be a number or a line or plot or a so i wonder like how we can really touch some deep truth in this in a subjective way like experience it really feel the beauty of it you know in the way that humans feel beauty right are we screwed i don't think we're screwed i think that we get little hints of it from from psychedelic drugs and so forth we get hints that there are certain interventions that we can take on our interface i apply this chemical which is just some element of my interface to this other to a brain i ingest it and all of a sudden i seem like i've opened new portals into conscious experiences well that's very very suggestive that's like um the black body radiation doing something that we didn't expect right it doesn't go to infinity when we thought it was going to go to infinity and we're forced to propose these quanta so once we have a theory of conscious agents and this projection is based on i should say i should sketch what i think that projection is um but then i think we can then start to ask specific questions when when you're taking dmt or you're taking lsd or something like that now that we have this deep model that we've reverse engineered space and time and physical particles we've pulled them back to this theory of conscious agents now we can ask ourselves in this idealized future um what are we doing to conscious agents when we apply 5mo dmt what are we doing are we opening a new portal right so when i say that i mean i have a portal into consciousness that i call my body of lex friedman that i'm creating and it's a genuine portal not perfect but it's a genuine portal i'm definitely communicating with your consciousness and we know that we have one technology for building new portals we know one technology and that is having kids having kids is how we build new portals into consciousness it takes a long time can you elaborate that oh oh oh you mean like your son and your daughter didn't exist that was a portal though you're having contact with consciousness that you never would have had before but now you've got a son or a daughter you had you went through this physical process they were born then you there was all the but is that portal yours so when you have kids are you creating new portals that are completely distinct from the portals that you've created with other consciousness like can you can you elaborate on that to which degree is are the consciousness of your kids a part of you well so every person that i see that symbol that i see the body that i see is is a portal potentially for me to interact with with a consciousness yeah um and and each consciousness has a unique character and we call it a personality right and so forth so with each new kid that's born we come in contact with a personality that we've never seen before and a version of consciousness that we've never seen before at a deeper level as i said the theory says there's one agent so this is a different projection of that one agent but but so that's what i mean by a portal is within my own interface my own projection can i see other projections of of that one consciousness so can i get portals in in that sense and i so and i think i think we will get a theory of that that we will get a theory of portals and then we can ask how the psychedelics are acting are they actually creating new portals or not if they're not we should nevertheless then understand how we could create a new portal right maybe we have to just study what happens when we make when we have kids we know that that technology creates new portals so we have to reverse engineer that and then say okay could we somehow create new portals de novo with that uh like uh brain computer interfaces for example yeah so maybe just a chemical or something else probably more complicated than a chemical that's why i think that the psychedelics the psychedelics may because they might be affecting this portal in certain ways that it turns it around and opens up in other words it may be once we understand what this thing is important your body as a portal and understand all those complexities maybe we'll realize that that portal can be shifted and into different parts of the the deeper consciousness and give new windows on it and so in that way maybe yes psychedelics could open up new portals in the sense that they're taking something that's already a complex portal and just tweaking it a bit well but creating is a very powerful difference between morphing right right tweaking versus creating i agree but maybe it gives you intuition to at least the full space of the kinds of things that this particular system is capable of i mean i mean the idea your idea the consciousness creates brains i mean that breaks my brain because because i you know i'm i'm i guess i'm still a physicalist in that sense because that you could it's just much easier to intuit the world um it's very it's practical to think all right there's a neural network and what are the different ways fascinating uh capabilities can emerge from this neural network uh it's free it's easier and so you start to and then present to yourself the problem of okay well how does consciousness arise how does intelligence arise how does emotion arise how does memory arise in the how do we filter within the system all the incoming sensory information we're able to uh allocate attention in different interesting ways how do all those mechanisms arise to say that there's other fundamental things we don't understand outside of space-time that are actually core to how this whole thing works is uh is a bit paralyzing because it's like oh we're not we're not ten percent done we're like zero point zero zero one percent done it's the f is it's the immediate feeling certainly understand that my attitude about it is if you look at the young physicists who are searching for the instructors beyond space time like amplitude and so forth they're having a ball space time that's what the old folks did that's what that's what our the older generation did we're we're doing something that really is fun and new and and they're having a blast and they're finding all all these new structures so so i i think that we're going to um succeed in getting a new deeper theory i can just say what i'm hoping with the theory that i'm working on i'm hoping to show that i could have this timeless dynamics of consciousness no entropic time i take a projection and i show how this timeless dynamics looks like the big bang and the entire evolution of space-time in other words i see how my whole space time interface so not just the uh the projections just doesn't just look like space-time you can explain the whole with it the whole from the origin of the universe that's that's what we have to do and that's what the physicists understand when they go beyond space time to the amplitude and the cosmological polytube they ultimately know that they have to get back the big bang story and the whole evolution that whole story where there were no living things there was just a point and then the explosion and then just particles at high energy and then eventually the cooling down and the differentiation and finally matter condenses and then life and then consciousness that whole story has to come out of something that's deeper and without time and that's what what we're up to that's we we want to to get this so the whole story that we've we've been telling ourselves about big bang and how brains evolved in unconsciousness will come out of a much deeper theory and and for yeah for someone like me um it's a lot i mean but for the younger generation this is like oh wow um all the low cherries aren't picked this is really good stuff this is really new fundamental stuff that we can do so that i can't wait to read the papers of the of the younger generation and i want to i want to see them uh kids these days with their non-space time assumptions uh it's just interesting looking at the philosophical tradition of this difficult ideas you struggle with if you look like somebody like emmanuel kant what are some interesting agreements and disagreements you have uh with a guy about the nature of reality so there's a lot in agreement right so kant was an idealist transcendental ideas and he he basically had the idea that um we don't see nature as it is we impose the structure on nature he and and so in some sense i'm saying something similar i'm saying that by the way i don't call myself an idealist i call myself a conscious realist because idealism has a long history a lot of different ideas come under idealism and there's a lot of debates and so forth it this tends to be identified with in many cases anti-science and anti-realism and i don't want either connection with my ideas and so i just called mine conscious realism with an emphasis on realism and not and not anti-realism but but the one place where i would of course disagree with kant was that he thought that um euclidean space time was a priori right we just know that that's false so so he he went went to too far on that but but in in general the idea that we don't start with space time that space and time is in some sense forms of our perceptions yes absolutely and i would say that you know there's a lot in common with barclay in that regard there's a lot of ingenious arguments in barclay leibniz leibniz in his monodology understood very clearly that the hard problem was not solvable he posed a hard problem and basically dismissed it and just you can't do this and so if he came here he and saw where we are he said look guys i told you this 300 years ago and he had his momentology he was trying to do something like this it's um it's different from what i'm doing but he had these things that were not in space and time the these moan ads he was trying to build something um i'm trying to build a theory of conscious agents my guess is that if he came here i could just if he saw what i was doing he would say he would understand it and immediately take off with it and go places that i couldn't he he would he would so there would be right there would be overlap of uh the spirit of the ideas absolutely totally overlapping but his genius would then just run with it far faster than i could i love the humility here so let me ask you about sort of practical implications of your ideas to our world our complicated world when you look at the big questions of humanity of hate war what else is there evil maybe there's the positive aspects of that of meaning of love um what is the fact that reality is an illusion perceived what what is the conscious realism when applied to daily life what kind of impact does it have a lot and it's it's sort of scary um we all know that life is ephemeral and spiritual traditions have said wake up to the fact that you know anything that you do here is going to disappear but it's even more ephemeral than perhaps we've thought i see this bottle because i create it right now as soon as i look away that data structure has been garbage collected that bottle i have to recreate it every time i look so i spend all my money i buy this fancy car that car i have to keep recreating it every time i look at it it's that ephemeral so all the things that we invest ourselves in we fight over we kill each other over and we have wars over these are all it's just like people in a virtual reality simulation right and and there's this this porsche and we all see the porsche well where that porsche exists when i look at it i turn my headset and i look at it and and then if joe turns his headset in the right way he'll see his porsche it's not it's not even the same portion that i see he's creating his own porsche so these things are exceedingly ephemeral and and now just imagine saying that that's my porsche well you can agree to say that it's your portion but but really the porsche only exists as long as you look so so this all of a sudden what the spiritual traditions have been saying for a long long time this gets cashed out in in mathematically precise science it's saying in femoral yes in fact it lasts for a few milliseconds a few hundred milliseconds while you look at it and then it's gone so so the whole idea why are we fighting why do we hate it's we fight over possessions because we we think that we're small little objects inside this pre-existing space time we assume that that that mansion and that car exists independent of us and that somehow we these little things can have our sense of self and importance enhanced by having that special car or that special house or that special person when in fact is just the opposite you create that mansion every time you look that's that's you're the you're something far deeper than that mansion you're the entity which can create that mansion on the fly and there's nothing there there's nothing to the mansion except what you create in this moment so all so all of a sudden when you take this point of view it has all sorts of implications for how we interact with each other how how we treat each other um and again a lot of things that spiritual traditions have said it's a mixed bag spiritual traditions are a mixed bag so let me just be right up front about that i'm not promoting any particular but they do have some insights yeah they have wisdom they have certain wisdom they have i can point to nonsense i won't go into it but i can also point to lots of nonsense so so the the issue is to then to look for the key the key insights and they i think they have a lot of insights about the ephemeral nature of of objects in space and time and not being attached to them including our own bodies and reversing that i'm not this little thing a little consciousness trapped in the body and the consciousness itself is only a product of the body so when the body dies the consciousness disappears it turns completely around the consciousness is fundamental the body my hand exists right now because i'm looking at it my hand is gone i have no hand i have i have no brain i have no heart if you looked you'll see a heart whatever i am [Music] is this really complicated thing in consciousness that's that's what i am all the stuff that i thought i was is something that i create on the fly and delete so this whole so this is completely a radical restructuring of how we think about possessions about identity about survival of death and and so forth this is completely transformative but the nice thing is that this whole approach of conscious stations unlike the spiritual traditions which have said in some cases similar things they set it in precisely this is mathematics we can actually now begin to stay precisely here's the mathematical model of consciousness conscious agents here's how it maps onto spacetime which i should sketch really briefly and here's why things are ephemeral and here's why you shouldn't be worried about the ephemeral nature of things because you're not a little tiny entity inside space and time quite the opposite you're the author of space and time the i and the am and the i am is all kind of emerging through this whole process of evolution and so on that's that's just surface waves and there's a much deeper ocean that we're trying to figure out here so how does you said this said this some of the stuff you're thinking about maps the space time how does it map the space right so so just a very very high level and i'll keep it brief the structures that the physicists are finding like the amplitude it turns out they're just static structure they're polytopes but they remarkably most of the information in them is contained in permutation matrices so it's a matrix like an n by n matrix that just has zeros and ones that contains almost all of the information and you can they have these plabic graphs and so forth that they use to boot up the scattering you can compute those scattering amplitudes almost entirely from these permutation matrices so that's just now from my point of view i have this conscious agent dynamics it turns out that the stationary dynamics that i was talking about the the where the entropy is increasing all the stationary dynamics are sketched out by permutation matrices so if you there's so-called burkhoff polytope all the vertices of this polytube all the points are permutation matrices all the internal points are markovian kernels um that have the uniform measure as a stationary measure i need to intuit a little better with what the heck you're talking about but so basically there's some complicated thing going on with the network of conscience conscious agents and that's mappable to this you're saying a two-dimensional matrix that uh uh scattering has to do with what with our perception like that's like photon stuff or i mean i don't know if it's useful to sort of uh dig into detail i'll do just a high level thing yes so the the the high level is the long-term behavior of the conscious agent that makes so that's the projection of just looking at the long-term behavior i'm hoping will give rise to the amplitude the amplitude then gives rise to space-time so then i can just use their link to go all the way from consciousness through its asymptotics to through the amplitude into space time and get the map all the way into our interface and that's why you mentioned the permutation majors because it gives you a nice thing to to try to generate that's right it's the connection with the amplitude the permutation matrices are the core of the amplitude and it turns out they're the core of the asymptotic description of the conscious agents so not to sort of bring up the idea of a creator but i i like first of all i like video games and you mentioned this kind of simulation idea first of all do you think of as an interesting idea this thought experiment that will live in a simulation and in general do you think we live in a simulation so the nick bostrom's idea about the simulation is typically couched in a physicalist framework yes so there's the bottom level there's some programmer in this physical spacetime and they have a computer that they've programmed really cleverly where they've created conscious entities so you have the hard problem of consciousness right the standard hard problem how could a computer simulation create a constant which isn't explained by that simulation theory but then the idea is that the next level the the entities that are from the that are created in the first level simulation then can write their own simulations and you get this this nesting so so the idea that um this is a simulation is fine but the idea that this starts with a physical space i think isn't but there's there's different properties here the the partial rendering and to me that's the interesting idea is not whether the entirety of the universe is simulated but how efficiently can you create interfaces that are convincing to all other entities that can appreciate such interfaces how little does it take because you said like partial rendering or like temporal affirmative rendering of stuff only render the tree falling in the forest when there's somebody there to see it it's interesting to think how can you do that super efficiently without having to render everything and that to me is one perspective on the simulation just like it is with video games right where a video game doesn't have to render every single thing it's just the thing that the observer is looking at right there is actually that's a very nice question and there's whole groups of researchers that are actually studying in virtual reality what what is the sort of minimal requirements on this system what how does it have to operate to give you an immersion experience to give you the feeling that you have a body to to get you to take it real and there's actually a lot of really good work on that right now and it turns out it doesn't take that much you do need to get the perception action loop tight and and you have to give them the perceptions that they're expecting if you want them to but if you you you can lead them along if you give them perceptions that are close to what they're expecting you can then maybe move their reality around a bit yeah it's a tricky engineering problem especially when you're trying to create a product that costs little but that's i it feels like an engineering problem not a deeply scientific problem or meaning obviously it's a scientific problem but as a scientific problem it's not that difficult to trick us uh descendants of apes but here's here's a case for just us you know our own this is a virtual reality that we're experiencing right now so here's something you can try for yourself if you just close your eyes and look at your experience in front of you to be aware of your experience in front of you what you experience is just like a modeled dark gray but there's all sort of there's some dynamics to it but it's just dark gray but now i ask you instead of having your attention forward put your attention backward what is it like behind you with your eyes closed and there it's like nothing it's real so what is going on here what what am i experiencing back there right well it's it's i i don't know if it's nothing it's it's like i guess it's the absence of it's not even like darkness or something it's it even it's not even darkness there's no there's no qualia and yet there is a sense of being and that's the interesting thing there's a sense of being back so i close my if i put my attention forward i just i have the quality of a gray model thing but when i put my attention backward there's no quality at all but there is a sense of being yeah i i personally uh now you haven't been to that side of the room i have been to that side of the room so for me memories i start um i start playing the engine of memory replay which is like i i take myself back in time and think about that place where i was hanging out in that part that's what i see when i'm behind so which that's an interesting quirk of hum humans too we're able to we're collecting these experiences and we can replay them in interesting ways whenever we feel like it and it's almost like being there but not really but almost that's right and yet we can go our entire lives on this you're talking about the minimal thing for vr we can go our entire lives and not realize that all of my life it's been like nothing behind me yeah we we we're not even aware that all of our lives if you just just for the just pay attention close your eyes pay attention to what's behind me we're like oh holy smoke it's totally scary i mean it's like nothing there's no quality there at all how did i not notice that my entire life we're so immersed in the simulation we buy it so much yeah i mean uh you could see this with with children right with persistence you know you could do the peekaboo game you can hide from them and appear and they're fully tricked and in the same way we're fully tricked there's nothing behind us and we assume there is that's really interesting these theories are pretty heavy you as a human being as a mortal human being how has these theories been to you personally like are there good days and bad days when you wake up and look in the mirror and the fact that you can't see anything behind you the fact that it's rendered like is there interesting quirks you know you know nietzsche with his if you gaze long into the abyss the abyss gazes into you um how's this theories these ideas change you as a person it's been very very difficult this stuff is not just abstract theory building because it's about us sometimes i realize that there's this big division of me my my mind is doing all this science and and coming up with these conclusions and the rest of me is not integrating i was just like i don't believe it i just don't believe this i mean it seems so as i start to take it seriously it's get i get scared myself it's like but it's very much then i read these spiritual traditions and realize they're saying very very similar things it's like there's a lot of conversions so for me i have the first time i thought it might be possible that we're not seeing the truth was in it was from some mathematics we were doing and when that hit me it hit me like a ton of bricks i had to sit down it was it it really it was scary it was really a shock to the system and then to realize that everything that has been important to me like you know getting a house getting a car getting a reputation and so forth well that car is just like the car i see in the virtual reality it's there when you perceive it it's not there so the whole question of you know what am i doing and why what what's what's worthwhile doing in life clearly getting a big house and getting a big car i mean we all knew that we were going to die so we we we we tend not to know that we tend to hide it especially when we were young before age 30 we don't believe we're going to die yeah we factually maybe know that you're you kind of are supposed to yeah but but they'll figure something out and yeah we'll be the generation that is the first one that doesn't have to die that's the kind of thing but but when you really face the fact that you're going to die and then when you when i start to look at it from this point of view that well this thing was an interface to begin with so what i'm really is what i'm really going to be doing just taking off a headset so i've been playing in a virtual reality game all day and i i got lost in the game when i was fighting over a porsche and i i shot some guys up and i punctured their tires and i got the porsche now i take the headset off and what was that for nothing there was just it was a data structure and the data structure is gone so so all of the wars the fighting and the reputations and all this stuff you know where it's just a headset so now and you know so my theory says that intellectually my my mind my my emotions rebel all over the place this is like i you know and so so i have to imagine i meditate a lot well what percent of the day would you say you spend as a physicalist um sort of living life pretending your car matters your reputation matter like like how much uh was that tom wait song i like my town with a little drop of poison how much poison do you allow yourself to have i think my default mode is physicalist right i think that that's just the default i i when i'm not being conscious yeah consciously attentive intellectually consciously because if you're just you're still if you're tasting coffee and not thinking or drinking or just taking in the sunset you're not being intellectual you're but you're still experiencing it right so it's when you turn on the like the introspective machine that's when you can start and turn off the thinker when i actually just start looking without thinking so that's that's when i feel like i all of a sudden i'm starting to see through sort of like okay part of part of the addiction to the interface is all the stories i'm telling about it's really important for me to get that really important to do that all so i'm telling all these stories and so i'm all wrapped up almost all the mind stuff that's going on in my head is about attachment to the interface and so what i found is that the essentially the only way to really detach from the interface is to [Music] literally let go of thoughts altogether and then all of a sudden um even my identity of you know my whole history my name my education all this stuff is almost irrelevant because it's just now here is the present moment and this is this is the reality right now and all of that other stuff is an interface story but this conscious experience right now this is the only this is the only reality as far as i can tell the rest of it's a story and but that is again not my default that is i have to make a really conscious choice to say okay i know intellectually this is all an interface i i'm going to take the headset off and so forth and and and then immediately sink back into the game and just be out there playing the game and get lost in so i'm always lost in the game unless i literally consciously choose to stop thinking isn't it terrifying to acknowledge that to look beyond the game isn't it uh scares the hell out of me it really is scary because i'm so attached i'm attached to this body i'm attached to the interface are you ever worried about breaking your brain a bit meaning like it's uh i mean some of these ideas when you think about reality even with like einstein just realizing you said interface just realizing that light you know that there's a speed of light and you can't go fast in the speed of light and like what kind of things black holes and can do with light even that can mess with your head yes but that's still space time that's a big mess but it's still just space-time it's still a property over interface that's right but it's still like even so even einstein realized that this particular thing some of the stories we tell ourselves is constructing interfaces that are oversimplifying the way things work because the it's nice the stories are nice stories are nice this rep i mean just like video games they're nice right and but einstein was a realist right he was a famous realist in this in the sense that he he was very explicit in a 1935 paper with um podesta and rosen the epr paper where he they said if without in any way disturbing a system i can predict with probability one the outcome of a measurement then there exists in reality that element right that that value that and we now know from quantum theory that that that's false that einstein's idea of local realism is is strictly speaking false yeah and and so we can predict we can set up in quantum theory you can set up and there's a there's a paper by chris fuchs quantum basianism where he he scouts this out it was done by other people but he gives a good presentation of this where they have a sequence of like something like nine different quantum measurements that that you can make and you can predict with probability one what a particular outcome will be but you can actually prove that it's impossible that the value existed before you made the measurement so you know with probability one what you're going to get but you also know with certainty that that value was not there until you made the measurement so the so we know from quantum theory that the act of observation is an act of fact creation and that is built into what i'm saying with this theory of consciousness if consciousness is fundamental space time itself is an act of fact creation it's it's an interface that we create consciousness creates plus all the objects in it so local realism is not true quantum theory is established also non-contextual realism is not true and that that it fits in perfectly with this idea that consciousness is fundamental these things are these exist as data structures when we create them as as as chris says the act of observation is an active fact creation but i must say on a personal level i'm having to spend i spend a couple hours a day just sitting in meditation on this and facing the rebellion in me that goes to the co it feels like it goes to the core of my being rebellion against these ideas so so here is very very interesting for me to look at this because so here i'm a scientist and i'm a person the science is really clear local realism is false non-contextual realism is false space time is doomed it's very very clear it couldn't be clear and my my emotions rebel left and right when i sit there and say okay i am not something in space and time then something inside of me says you're crazy of course you are and i'm completely attached to it i'm completely attached to all this stuff i'm attached to my body i'm attached to the headset i'm attached to my car attached to people i'm attached to all of it and and yet i know as a absolute fact i'm going to walk away from all of it i'm going to die it'll you know in fact i almost died last year i mean covet almost killed me it i i i sent a goodbye text to my wife so i was i thought you really did i sent her a goodbye i thought i was in the emergency room and uh it had attacked my heart and it been at 190 beats per minute for 36 hours like i couldn't last much longer i knew i could they couldn't stop it so that was that was it so that was it so so i texted her goodbye from the emergency room i love you goodbye kind of thing yeah right yeah that was it so so were you afraid yeah i was scared to help you right but there there is there was you're just feeling so bad anyway that that all you know you that's sort of what you're scared but you're just feeling so bad that in some sense you just wanted to stop anyway yeah so so i've i've been there and faced it just just a year ago how did that change you by the way having having this intellectual reality that's so challenging that you meditate on that you're it's just an interface and one of the one of the hardest things to come to terms with is that that means that you know it's going to end um how did i change you having come so close to the reality of it it's not just an intellectual reality it's it's a reality of death it's it's forced i've meditated for 20 years now and then i would say averaging three or four hours a day um but it's put a new urgency but is it urgency is not the right word because that that it's it it's riveted my attention i'll put it that way it's really riveted my attention and um i've really paid i spent a lot more time looking at what spiritual traditions say i don't by the way again not taking it with the you know i take it all with a grain of salt but on the other hand i think it's stupid for me to ignore it so i try to listen to the best ideas and and to sort out nonsense from and it's just we all have to do it for ourselves right it's not easy so what makes sense and i have the advantage of some science so i can look at what science says and try to compare with spiritual tradition i try to sort it out for myself and but then i also look and realize that there's another aspect to me which is this whole emotional aspect the i i seem to be wired up as evolutionary psychology says i'm wired up right all these defensive mechanisms you know i'm inclined to lie if i need to i'm inclined to to be angry to protect myself to have an in-group and an out group to try to make my reputation as big as possible to try to demean the out group there's all these things that evolutionary psychology is is spot on it's really brilliant about the human condition and yet i think evolution as i said evolutionary theory is a projection of a deeper theory where there may be no competition so how so i'm in this very interesting position where i feel like okay according to my own theory i'm consciousness and maybe this is what it means for consciousness to wake up it's not easy it's it's it's almost like i have i feel like i have real skin in the game it really is scary i really was scared when i was about to die it really was hard to say goodbye to my wife it really it really pained and to then look at that and then look at the fact that i'm going to walk away from this anyway and it's just an interface how do i so it's it's trying to put all this stuff together and really grok it so to speak not just intellectually but rocket at an emotion yeah what are you afraid of you silly evolved organism that's gotten way too attached to the interface what are you really afraid of that's right is there uh very personal you know it's very very personal yeah yeah i mean speaking of the text what do you think is uh this whole love thing what's the role of love in our human condition this interface thing we have this is this somehow interweaved interconnected with consciousness this attachment we have to other humans and this deep like some um there's some quality to it that seems very interesting peculiar well there are two levels i would think about that there there's love in the sexual sense and there's love in a deeper sense and in the sexual sense um we can give an evolutionary account of that and and so forth and i think that's pretty clear to people um in in in this deeper sense right so of course you're married you're i love my wife in a sexual sense but there's a deeper sense as well that when i was saying goodbye to her there was a there was a deeper much deeper love that was really at play there that's one place where i think that the mixed bag from spiritual traditions has something right when they say you know love your neighbor as yourself that that some in some sense love is fundamental i think that they're on to something something very very deep and profound and every most of all i can get a personal glimpse of that when i especially when i'm in the space with no thought right when when i can really let go of thoughts i get little glimpses of of a love in the sense that i'm not separate it's a it's a love in the sense that i'm not different from that i you know yeah if you are separate then then there's i can fight you but if you and i are the same if there's a union there the togetherness of it yeah what what uh who's god all those gods the stories that been told throughout history you said through the spiritual traditions what do you think that is is that us trying to find that common thing at the core oh well in in many traditions not all the one i was raised in so my dad was a protestant minister we tend to think of god as a being but i think that that's not right i think the closest way to think about god is being period not a being but being the very ground of being itself is god i think that's the the deep and from my point of view that's the ground of consciousness so the ground of conscious being is what we might call god but but the word god has always been you know for example you don't believe the same god as my god so i'm going to fight you or i will we'll have wars over because the being the specific being that you call god is different from the being that i call god and so we fight whereas if it's not a being but just being and you and i share being then you and i are are not separate and there's no reason to fight we're both part of that one being and and loving you is loving myself because we're all part of that one being the spiritual traditions that point to that i think are pointing it in a very interesting direction and that does seem to match with the mathematics of the conscious agent stuff that i've been working on as well that it really fits with that although that wasn't my goal is there uh you mentioned you mentioned that the young physicists that um you talk to or whose work you follow have quite a lot of fun breaking with the traditions of the past the assumptions of the past uh what advice would you give to young people today in high school and college not just physicists but uh in general how to have a career they can be proud of how they can have a life they can be proud of how to make their way in the world from the lessons from the wins and the losses in your own life what what what little insights could you pull out i would say the universe is a lot more interesting than you might expect and you are a lot more special and interesting than you might expect you might think that you're just a little tiny irrelevant 100 pound 200 pound person in a vast billions of light years across space and that's not the case you are in some sense the being that's creating that space all the time every time you look so waking up to who you really are outside of space and time as the author of space and time is the author of everything that you see the author of space and time sorry you're the author of space and time right and i'm the author of space and time and space and time is just one little data structure many other consciousnesses are creating other other data structures they're authors of various other things so so realizing and then realizing that that um i had this feeling that growing up and going and called reading all these textbooks oh man it's all been done yeah if i'd just been there you know 50 years ago i could have discovered this stuff but the you know it's all in the textbooks now well believe me the textbooks are going to look silly in 50 years and it's your chan your chance to write the new textbook so so of course study the current textbooks you have to understand them you there's no way to progress until you understand what's been done but but then the only limit is your imagination frankly that's the only limit the greatest books the greatest textbooks ever written on earth are yet to be written exactly uh what do you think is the meaning of this whole thing what's the meaning of life from your limited interface can you can you can you figure it all out like why why so you said the universe is kind of trying to figure itself out through through us uh why why yeah that's the closest i've come so i'll give you so i will say that i don't know but but but i'll here's my guess right that's a good first sentence that's a good starting point and and maybe that's going to be a profound part of the final answer is to start with the i don't know it's quite possible that that that's really important to start with the i don't know my guess is that if consciousness is fundamental and of girdle girdles and completeness theorem holds here and there's infinite variety of structures for consciousness to some sense explore um that maybe that's what it's about this is something that anika and i talked about a little bit and she doesn't like this way of talking about it so i'm gonna have to talk with some more about this way of talking but right now i'll just put it this way and i'll have to talk with her more and see if i can say it more clearly but the way i i'm talking about it now is that there's a sense in which there's being and then there's the experiences or forums that come out of being that's one deep deep mystery and the the question of that you asked what is it all about somehow it's related to that why does being why does it just stay without any forms why does it why don't why do we have experiences what why should why why not just have when you close your eyes and you pay attention to what's behind you there's nothing but there's being why is why don't we just stop there why didn't we just stop there why did we create all tables and chairs and the sun and moon and people that all this really complicated stuff why and and all i can guess right now and i'll probably kick myself in a couple years and say that was dumb but but all i can guess right now is that somehow consciousness wakes up to itself by knowing what it's not so here i am i'm not this body and i sort of saw that it was sort of in my face when i sent a text goodbye but then as soon as i'm better it's sort of like okay i i sort of don't want to go there right okay so i just so i am my body you know about my go back to the standard thing i own my body and you know and then i want to get that car and even though i was just about to die a year ago so that comes rushing back so so consciousness immerses itself fully into a particular headset gets lost in it and then slowly wakes up just so it can escape and that is the waking up but he needs to have needs to know what is not it needs to to know what you are you have to say oh i'm not that i'm not that that wasn't important that wasn't important that's really powerful don let me just say that um because i've been a long term fan of yours and we're supposed to have a conversation during this very difficult moment in your life let me just say you're truly special person and i for one and i know there's a lot a lot of others that agree i'm glad that you're still here with us on this earth if for a short time um so whatever um whatever the universe this whatever plan it has for you that brought you close to death to maybe enlighten you some kind of way um i think i think he has a has an interesting plan for you you're one of the truly special humans and it's a huge honor that you would sit and talk with me today thank you so much thank you very much alex i really appreciate that thank you thanks for listening to this conversation with donald hoffman to support this podcast please check out our sponsors in the description and now let me leave you with some words from albert einstein relevant to the ideas discussed in this conversation time and space are modes by which we think and not conditions in which we live thank you for listening and hope to see you next time

Channel: Lex Fridman

Views: 5,672,418

Keywords: agi, ai, ai podcast, artificial intelligence, artificial intelligence podcast, consciousness, donald hoffman, evolution, game theory, lex ai, lex fridman, lex jre, lex mit, lex podcast, mit ai, reality, space, spacetime

Id: reYdQYZ9Rj4

Length: 196min 16sec (11776 seconds)

Published: Sun Jun 12 2022

Reddit Comments

[u/ringo-san]

Wow, I just finished reading On the Origin of Species and so this podcast is right on time. Evolution says that what we see and are able to understand is only what we need to see and understand in order to be able to out compete the monkeys that came before us. It's such hubris to think we can know what is fundamental in the universe.

[u/Elecrockcity]

This dude could be making shit up on the spot and I’d have no fuckin idea.

[u/infinitejesttt]

Been hoping for this one for a while

[u/cpt_mojo]

I love most of Lex's episodes, but I honestly found this one a bit frustrating to listen to.

Donald had for me too many hand-wavy arguments that left me often with doubt and questions.

I have a Masters in Physics, PhD in Biophysics and now work in Data Science with statistics and optimisation like Genetic Algorithms. So, I believe I understand a thing or two, but maybe this episode just went over my head.

For example, very fundamentally, Donald's work itself must be an illusion, following its own logic, so why would it be more real than anything else. Or in other parts, the conversation was simply about advancement in science, peeling the onion of truth for which the illusion theory is irrelevant.

Anyone else in the same boat? I'd love to read/hear about scientific reviews of Donald's work and what of it is controversial in the field.

Edit: also the topic on probability of zero, but still possible that the event happens - I found this odd, and Lex also called it out. But then Donald answers hand-wavy and follows with basically: "you have to be a mathematician to explain this, and I'm not". It just felt too convenient...

[u/TheMaskedBallsack]

Kant sheds a lot of light on this discussion. Lex should have more philosophers on the show.

Edit: just got to the part where they discuss Kant lol

[u/TheSeaBast]

Wowzers. Favorite episode I've listened to so far. This is an idea really similar to one I've thought about a lot after being exposed to it from buddhist teachings, but never knew was actually considered in the academic sphere. Can't wait to dig more into Hoffman and this theory.

[u/kromslaugh]

I was excited but I don't understand anything he is saying so I had to shut it off

[u/asmdsr]

Donald Hoffman is a crackpot. I don't know why so many people here are swallowing his bullshit.

The most charitable way I can describe Hoffman is that he is working with a parallel epistemology, using the same words we use for different things (e.g. reality, objective, real, true, fundamental, etc) and he doesn't understand emergence. And at the end of the day, his theories have no relation to consciousness / subjective experience (explain nothing, predict nothing about it).

His evolution argument is proven false by the many scientific observations we have made using instruments that were not evolved through natural selection, yet agree with the observations made via our perception.

As for his interface theory:

Similarly, the claim of Interface Theory is that perceptual properties of space-time and objects simply reflect characteristics of our perceptual interface; they do not correspond to objective truth

Same argument, we have overwhelming evidence from non-perceptive scientific instrument that reveal an extremely high correspondence with a space-time reality independent of our minds (aka objective reality). If the only argument is that space-time is not fundamental, well physicists have been telling us that for decades - which makes space-time emergent, but not an illusion. And here the debate just devolves into semantics.

[u/UncleWeyland]

Good episode, even though I would have pushed back hard on a lot of stuff.

When he steel-man's the physicalist position, I would have opted for a very different tack.

Consider that changes in consciousness seem to be completely supervenient on physical manipulations of the brain. Tononi makes the case very lucidly and forcefully in Phi.

If consciousness is fundamental, and you build up the spacetime "desktop and icons" from it, then why is it that when I drag the "probe" icon into your "brain" icon, I can elicit qualia, memories, actions and even false rationalizations/confabulations in your subjective experience which you use to incorrectly causally explain the source of your arm movements?

Hope you get a chance to talk with him again before too long! (And maybe book Tononi, just don't ask him about Aaronson's critique he gets touchy about it).