Gödel, Escher, Bach author Doug Hofstadter on the state of AI today

it hit me like a ton of bricks all of a sudden how a brain a physical object inside our head is responsible for all that we consider ourselves our feelings our souls everything about us and it led me to asking all sorts of questions about how it was possible for physical object to support something so abstract and ineffable as a self or a soul or an eye Doug Hostetter is a cognitive scientist and professor at Indiana University but he's best known as the author of The Pulitzer prize-winning book goodle Escher Bach for me and for many people in my generation gev as we affectionately called it was a landmark work that brought together our mutual fascination with computational systems how the mind works and the beauty of paradox not to mention it drew connections between art and music and Mathematics all things that I love deeply thanks to my partner Scott Kim Doug has been part of my life for many years Scott and Doug first met back in 1975 when they were part of a circle of friends that nurtured the creation of Geb join us as we talk about the origins of Doug's interest in the mind how he came to be writing guttle Asher Bach and what he thinks about the recent wave of advancements in AI foreign so Doug how did you first get interested in Ai and cognitive science I wondered how it was that I created sentences in French as opposed to creating them in English and the bubbling up of ideas was something that fascinated me then I also admired enormously certain creative Geniuses and I wondered things about their minds how they did what they did my sister my youngest sister Molly had brain damage and I didn't think so much about phrases like brain damage or something but when my parents bought a book about the brain and I thought about Molly and I started reading this book it hit me like a ton of bricks all of a sudden how a brain a physical object inside our head is responsible for all that we consider ourselves our feelings our souls everything about us and it led me to asking all sorts of questions about how it was possible for physical object to support something so abstract and ineffable as a self or a soul or an eye and lastly when I was 14 or 15 I read a book called girdles proof by Ernest Nagel and James R Newman and that book was about the whole in a certain sense at the center of mathematics the idea of unprovable statements that were for the reason that they were unprovable is that in a certain unexpected way statements of mathematics could be made to twist around and talk about themselves and the legit the Austrian logician quote girdle in 1930-31 was able to create a statement that said essentially I am not provable within a certain formal system and for a statement to talk about itself to be able to talk about itself was just a miraculous thing to me and it opened all sorts of doors in my mind so it was a combination oh and one other thing a very crucial thing I learned a program when I was 15 from my friend Charlie Brenner and I started programming all sorts of things and I knew how computers worked because I was a programmer and and I created a program in a mid-60s that was able to create sentences random sentences that employed randomly chose chosen pathways through syntactic Network and randomly chosen words filling in the parts of speech a noun or a verb or an adverb or whatever and some of the sentences were very long and complex and very humorous some of them were not so humorous they were actually they sounded fairly meaningful and I that again made me think about what is it going on inside this computer that is similar to and what is you know different from what happens when I myself come up with sentences whether in French or in English or in any other language and so it was a combination of all those things computers my sister Molly my interest in languages girdles theorem and so many things that came together to make me interested in these questions right so you mentioned and an aha moment that had to do with recursion or self-reflection yeah can you go in a little bit on that because that's such a fundamental Concept in Geb and it's also a fundamental concept and argument going on in today's Ai and llms yeah well the self-reference in girdle's construction comes about in a way that's very surprising because in 19 roughly 10 to 13 Bertrand Russell and Alfred North White had two important philosophers created a work called principia Mathematica meaning the bases of mathematics and they tried to found mathematics in logic but since Russell had created a paradox that involved the set of all sets that don't contain themselves he knew that this Paradox was fatal to mathematics and so he wanted to create a system that could not talk about such things and so he created an idea that I'm not going to go into but he called it the theory of types that prevented sets from containing themselves prevented sentences from talking about themselves Etc and he thought that by banishing self-reference he was going to be able to create the fundamental basis of all of mathematics and he did this in conjunction with Alfred North Whitehead the thing that was amazing though was that girdle in when he was about 24 25 years old he came up with this idea that numbers can stand for things we know that they stand for things in all sorts of ways and that numbers can stand for symbols and so he could create a sentence that was about numbers but at the same time it could be read at a second level so that it was about symbols and it turned out that he figured out a way to map the entire structure of sentences in or formulas in principia Mathematica onto number and so the sentence that he created could be read on one level as a sentence about numbers but on a second level it could be read as a sentence about structures in the formal structures in principia Mathematica and it could never thereby be talking about such things as theorems and proofs and axioms and so forth and in fact the way this sentence says that I am approval it really says there does not exist a derivation of a certain formula in principia mathematic derivation meaning a proof and and then the certain formula that is talking about turns out to be Itself by virtue of the mapping that girl created between symbols and numbers so he wound up binding creating a self-referential sentence in the Fortress that Bertrand Russell had erected to banish self-reference from so it was an amazing thing and it it it sidestepped all of Russell's ideas it turned self-reference into something that was inevitable even despite the best efforts to banish it and that struck me as very magical and it reminded me of the idea that a brain seems to be something that is inanimate it seems to be made of inanimate in the sense that it's made of molecules which are inanimate that are just doing their chemical things and yet somehow out of it comes not only Life Link you know the ability to perceive the ability to react to the world but also the the ability to create a self model the ability to create the feeling of Consciousness so it's a kind of a second level of looking at the brain the brain at one level is just a physical object and on a second level it's something very magical Circle because it creates a thinking feeling conscious being and so I made the analogy between Consciousness and girdles construction and I tried to spell it out I did it a little bit better I think some years later in my book I'm a strange Loop that came out about 25 years later maybe wow so you've been pulling this thread for a while yes yeah well in fact the first time I was thinking about it was when I was 16 years old and so I didn't write Geb until I was in my early 30s so in fact it goes way back to about 1961. over 60 years that and that's such a great story Fortress and not opening up it's like a mathematical trojan horse that's right that's correct that's a good phrase for it I love bringing self-reference in despite the fact that it had been officially and it's totally banished and then it just took over so tell us the story of how GB came into life it's a Monumental effort to write any book let alone a book like that let alone get a publisher let alone have the cojones to just pull that together and put it in the world what was that Journey like what you know well not that one yeah it began I became a graduate student in mathematics in 1966 and I dropped out because I didn't I wasn't able to handle it it was too abstract and I took a jump into physics in 68 after two years of struggle against mathematics and in 68 I became a graduate student in physics and I struggled in physics and that's a long story complex which I don't want to go into but it was a very painful part of my life and it lasted for quite a number of years and I was interested in many things but I had long left behind my interests in computers in Consciousness in self-reference and girdle's Theorem all of those things I had left behind in going into physics but I loved moseying around the bookstore at the University of Oregon where I was and one day I came across a book called profile of mathematical logic by Howard DeLong and I picked it up out of curiosity I mean godel's theorem belongs to mathematical logic and so it reminded me of my old interests from quite a number of years earlier and I picked it up and started flipping through it and I got completely sucked in it was very rapid intoxication with that book and the book re-inflamed all of my passionate interests that I had as a teenager in self-reference and so forth and I couldn't stop thinking about it even though I was a physics student this would be in about 1972 and I could not stop thinking about these things and one day I started writing a letter to my friend Robert burninger it was a long letter I happened to be in Boulder Colorado and I was in the library of the University of Colorado and I was sitting at a big table and I had a bunch of paper and I wrote a letter that was 32 pages long that was put Ting forth some of my ideas about Consciousness mathematics abstract structures codes self-reference computers there were formal systems proofs so many things it's taken me three or four hours and I thought this is I think I can't go any further today but maybe I've done about half of what I need to do so I'll mail this letter off and maybe I'll write the other half of the letter in the near future well that 32-page letter was sort of The Germ of Geb and I didn't wind up writing another 32 pages but when I got back to Oregon several months later I wound up writing a draft of a book which at the time was called girdle's theorem in the human brain and that was the first title and I did it all very rapidly in the fall of 1973. I wrote this book maybe 200 pages in pen on just ordinary paper one day when I was thinking about a particular issue I started writing a dialogue that was modeled on a dialogue that Lewis Carroll had written called what the tortoise said to Achilles and I used the same characters I used the tortoise and achilles in my my own dialogue and they were very humorous characters and I was able to pick up their character traits and write an amusing dialogue and I thought this is fun I'll try to put this into my book and then I got into the frame of mind of writing more dialogues once in a while and I wrote two or three more and at one point I wrote a dialogue that was structurally kind of tricky and just for the fun of it I went back to the very beginning of the dialogue and I typed the word Fugue at the beginning it wasn't really very frugal but it reminded me vaguely of a Fugue and all of a sudden that writing of that one word sparked in my mind the idea that maybe I could write a dialogue that really was like a few or another kind of piece by Bach like a Canon c-a-n-o-n that is which is like a round in music but it could be more complex and I thought gee a dialogue with an interesting structural form as well as interesting ideas would be a novelty and so that became a second facet of the book writing dialogues that hit interesting structural forms that were based at first on Bach music and and so forth the structures became more and more elaborate and it eventually I wound up inserting intricately structured dialogue between every pair of consecutive chapters and that made the book have a very different flavor from a book called goodo serum in the human brain and I knew it had to have a different title and since contrapuntal music was playing a role in the book a very important role in determining the structures of the dialogues I decided Well girdle and Bach and then I also my dad had read an early draft of it and it critiqued it a bit and he had said a lot of things that were use useful to me but one of them he said was why aren't you have more pictures and then it occurred to me that in the back of my mind as I was writing a lot of the book there were pictures by MC Escher paradoxical strange pictures that were flooding through my brain as I was writing but I wasn't telling the readers about them I wasn't saying a thing about any pictures at all by esher or by anybody else and it occurred to me if my dad thinks I should have pictures why don't I include some Asher so then Escher came into the book and then I thought well this book is really full of art full of references to Art full of references to music in some ways and of course Google so why don't I just call it good old Usher that'll suggest to people that well of course to knowledgeable people actually Usher wasn't very well known and Google certainly wasn't known so it wouldn't necessarily suggest too much to people other than the word Bach and then I invented the subtitle an internal Golden Braid which was the same three letters egb in a different order and the idea started getting more and more self-involved and during that time while I was writing a third draft of the book at Stanford in 1975 76 77 I got to know Scott and Scott's way of writing had a big influence it's on me he was very playful in his use of language he loved to use parallel paragraphs he would write a paragraph that was talking about one thing and then he would write a paragraph that was almost identical but that was talking about something completely different and I thought that was very beautiful and it influenced quite a bit of things that I wrote In the book and I was spending huge amounts of time with Scott during the final the writing of the final version which was a say 75-77 so then I was lucky enough to be able to typeset my own book and so forth but those are separate stories I don't know how important they are wow so ledge yeah who wrote the text editing program that I used to write the book and also the typesetting program that I used to typeset it wow how did you find a publisher I was just pretty naive I wrote a cover letter I guess and I took a chapter or two and I just sent them out to a bunch of Publishers and mostly I got rejections of you know all the Publishers that I first thought of said they thought it was interesting or something like that but it wasn't their type of book but the 12th publisher as I recall that I sent it out to which was basic books was enthusiastic and I guess it was because they sent it out to a physicist named Jeremy Bernstein and Bernstein gave it an incredibly favorable review and I think it was thanks to Jeremy Bernstein perhaps also Freeman Dyson another physicist Freeman Dyson gave very positive comments as well it was Martin Kessler at basic books who sent it out to them he was the president and I believe it was because he got back such favorable reports from these very knowledgeable people that that the book was accepted by basic books and then what an unlikely hit well it was an unlikely hit maybe you're right I agree with you there but at the same time again I owe to Scott the fact that he wrote something called The Strange Loop Gazette because the concept strange Loop would represented this idea of self-reference that was at the core of godel's theorem and at the core a human eye and it was a term that I used in Geb quite often and especially toward the end of the book and Scott wrote The Strange Loop Gazette which was a several page document explaining a lot of the book to an idol that we shared namely Martin Gardner who wrote a monthly column in Scientific American called mathematical games and that letter from Scott if you wish to call it a letter it was more than a letter but anyway the strange Loop Gazette that Scott wrote and sent to Martin Gardner gotten Martin extremely excited about the book and he wrote an incredibly favorable review of the book and and that must have helped Propel the enormously the popularity and success he wrote that in July of 1979 and the book received the Pulitzer Prize and another award in the middle of the next year and certainly Martin Gardner's endorsement thanks to Scott I would say was pivotal wow that was another his column was another Touchstone for me when I was in school and we all looked forward to reading it whenever yeah well whenever Scientific American arrived in the mailbox the first thing I would flip to was about page 125 roughly and see what is what is Martin Gardner say this month what what an amazing guy and what an amazing story so what of all the ideas you explored in Geb which ones do you think are most relevant for today's budding AI scientists and enthusiasts well you know I think the question Still Remains what is an eye what is consciousness what exactly is thinking I think that many people are puzzled about whether computers especially I don't want to use the word computer since something like chat GPT is kind of much bigger system than what we usually call a computer but anyway a computational system I may say computers in the future because I slip but I really mean computational system whether such things made of very different Hardware from animal Hardware from human beings can have anything like experiences feelings thoughts ideas meaning in what they're saying there are certain people who are naysayers who say that everything that comes out of these kinds of systems like chat GPT is inherently meaningless and it's just symbols being battered about by systems that have no understanding of anything and I think that's a misleading and misled opinion part of what I learned when I was writing the program that created sentences back in the mid 60s you know I was wondering why how I was different from a computational system that was creating sentences and I felt that the essential difference was that the behind the words there wasn't meaning in the computational system and behind my words there was meaning and what was the difference what made something have meaning and I thought a long time about what made something have meaning and I talked about it a lot in Geb and I felt that it was when the symbols in a system in GP is full of formal systems they're not really exactly computational systems but they're similar it's this formal rules that guide symbols and make symbols work in certain ways it's when the symbols in that system are tracking something in the real world when they parallel something in the real world so exactly that you can say that they stand for those things in my sentence creation program my the words weren't tracking anything they were just being pushed around at random by programs that selected Pathways to a syntactic network and just selected words to fill in but the words were not being used because they had certain meanings they were just being shoved in at random and but when words are very systematically correlated with phenomena in a very coherent consistent way a over a long time you come to believe that those words or those symbols really can be said to have meaning and it seems that today's systems are doing that a great deal sometimes they fall all over their faces I mean I've recently saw a proof in quotes and proof by chat GPT that claimed to prove that every number of the form 3n plus one where n is an integer is odd which is crazy it's nonsense and you find a lot of nonsense still occasionally being produced by these chat Bots but it's being reduced over time and a lot of what they're producing is totally coherent and believable and sensible and so you have to or I have to I don't know of one in general that I have to start assigning meaning to the symbols that they're using and saying that if there's meaning here then there are ideas here and if there are ideas here then there's thinking here and if there is thinking then there is some degree of Consciousness here and it's a kind of a slippery slope and right now we don't know where we are on that slippery slope uh we don't understand very well so Geb is was trying to set forth and later I am a strange Loop was trying to set forth what it is that really makes a self or a soul I like to use the word Soul not in the religious sense but sort of a synonym for I a human eye capital letter I and so what is it that makes a human being be able to validly say I what is it that justifies the use of that word when can a computer say I and we feel that there is a genuine eye behind the scenes I don't mean like when you call up the drugstore and the chat bot I don't know if I should call it that but anyway the whatever you want to call it on the phone says tell me what you want I know you want to talk to a human being but first in a few words tell me what you want I can understand full sentences and then it you know you say something and it says do you want to refill a prescription and then I say yes it says gotcha meaning I got you so it acts as if there is an eye there but I don't have any sense whatsoever that there is an eye there it doesn't feel like an i in the least to me it feels like a very mechanical process but in the case of more advanced things like chat gpt3 or chat gpt4 it feels like there is something more there that merits the word I and the question is when will we feel that those things actually deserve to be thought of as being full-fledged or at least partly fledged eyes and I personally worry that this is happening right now but it's not only happening right now it's not just that certain things that are coming about are similar to human consciousness or human selves they are also very different and in one way it is extremely frightening to me they are extraordinarily much more knowledgeable and they are extraordinarily much faster so that if I were to take an hour in doing something the chat GPT 4 might take one second I don't know maybe not even a second to do exactly the same thing and that suggests that these entities whatever you want to think of them are going to be very soon right now they still make so many mistakes that we can't call them more intelligent than us but very soon they're going to be they may very well be more intelligent than us and far more intelligent than us and at that point we will be receding into the background in some sense we will be we will have handed the Baton over to our successors For Better or For Worse and I can understand that if this were to happen over a long period of time like hundreds of years that might be okay but it's happening over a period of a few years it's like a tidal wave that is washing over us at unprecedented and unimagined speeds and to me it's quite terrifying because it suggests that everything that I used to believe was the case is being overturned what are some of the things specifically that terrify you what are some issues that you're really I when I started out studying cognitive science and thinking about the mind and computation you know this was many years ago 1960 roughly and I knew how computers worked and I knew how extraordinarily rigid they were you made the slightest typing error and it completely ruined your program and debugging was a very difficult art and you might have to run your program many times in order to just get the bugs out and then when it ran it would be very rigid and it might not do exactly what you wanted it to do because you hadn't told it exactly what you wanted to do correctly and you had to change your program and on and on and on computers were very rigid and I grew up with a certain feeling about what computers can or cannot do and I thought that artificial intelligence when I heard about it was a very fascinating goal which is to make rigid systems act fluid but to me that was a very long remote goal it seemed infinitely far away it felt as if artificial intelligence was the art of trying to make very rigid systems behave as if they were fluid and I felt that would take enormous amounts of time you know I felt it would be hundreds of years before anything even remotely like a human mind would be asymptotically approaching the level of the human mind but from beneath I never imagined that computers would rival or let alone surpass human intelligence and in principle I thought they could rival human intelligence I didn't see any reason that they couldn't but it seemed to me like it was a goal that was so far away I wasn't worried about it but when certain systems started appearing maybe 20 years ago they gave me pause and then this started happening at an accelerating Pace where unreachable goals and things that computers shouldn't be able to do started toppling the defeat of Gary Kasparov by deep blue and then going on to go systems go program well systems that could defeat some of the best bow players in the world and then systems got better and better at translation between languages and then at producing intelligible in responses to difficult questions in natural language and even writing poetry and my whole intellectual edifice my system of beliefs it's a very traumatic experience when some of your most core beliefs about the world start collapsing and especially when you think that the human beings are soon going to be eclipsed it felt as if not only are my belief systems collapsing but it feels as if the entire human race is going to be eclipsed and left in the dust soon people ask me what do you mean by soon and I don't know what I really mean I don't have any way of knowing but some part of me says five years some part of me says 20 years some part of me says I don't know I have no idea but the progress the accelerating progress has been so unexpected so completely caught me off guard not only myself but many many people that there is a certain kind of Terror of an oncoming tsunami that is going to catch all of humanity off guard it's not clear whether that will mean the end of humanity in the sense of the systems we've created destroying us it's not clear if that's the case but it's certainly conceivable if not it's also it just renders Humanity a small a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us as incomprehensible to us as we are to cockroaches that's an interesting thought well I don't think it's interesting I think it's terrifying I hate it I think about it practically all the time every single day wow and it overwhelms me and depresses me in a way that I haven't been depressed for a very long time wow that's really intense you have a unique perspective so knowing you feel that way is very powerful how have llms large language models impacted your view of how human thought and creativity works of course it reinforces the idea that the human creativity and so forth come from the brain's Hardware there is nothing else than the brain's Hardware which is neural Nets but one thing that has completely surprised me is that these llms and other systems like them are all feed forward it's like it's the firing of the neurons is going only in One Direction and I would never have thought that deep thinking could come out of a network that only goes in One Direction out of firing neurons in Only One Direction and that does that doesn't to make sense to me but that just shows that I'm naive it also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing GED and writing on a strange loop I felt at those times quite a number of years ago that as I say we were very far away from reaching anything computational that could possibly rival us it was getting more fluid but I didn't think it was going to happen you know within a very short time and so it makes me feel diminished it makes me feel in some sense like a very imperfect flawed structure and compared with these computational systems that have you know a million times or a billion times more knowledge than I have and are a billion times faster it makes me feel extremely inferior and I don't want to say deserving of being eclipsed but it almost feels that way as if we all we humans unbeknownst to us are soon going to be eclipsed and rightly so because we're so so imperfect and so fallible we forget things all the time we confuse things all the time we contradict ourselves all the time you know it may very well be that that just shows how limited we are wow so let me keep going through the questions is there a time in our history as human beings when there was something analogous that terrified a lot of smart people inspired yeah you didn't even hesitate did you so what can we learn from that no I don't know caution but you know we may have already gone too far we may have already set the forest on fire I mean it seems to me that we've already done that I don't think there's any way of going back when I saw an interview with Jeff Hinton who was probably the most Central person in the development of of all of these kinds of systems firstly he said he might regret his life's work part of him is what he said he said part of me regrets all of my life's work and the interviewer asked him how important is this these are these developments are they as important as the Industrial Revolution and hint and thought for a second and he said well maybe as important as the wheel what brings me joy is clever bonmos quips spontaneous pieces of word play or jokes spoken by friends that you know it brings me some Joy seeing friends brings me joy you brought us a lot of Joy by being here today and sharing your perspective it's my pleasure really valuable great

Channel: Game Thinking TV

Views: 76,631

Keywords: game thinking, ai, machine learning, artificial intelligence, chatgpt, doug hofstadter, hofstadter, douglas hofstadter, amy jo kim, game design, innovation, entrepreneur, startup, startup advice, gamification, mind, self, self reference, recursion, geb, godel escher bach, how the mind works, cognitive science, strange loop, llm, large language model, consciousness

Id: lfXxzAVtdpU

Length: 37min 56sec (2276 seconds)

Published: Thu Jun 29 2023

Reddit Comments

[u/Smallpaul]

"The accelerating progress has been so unexpected and so completely caught me off guard. Not only myself but many, many people. There is a certain kind of Terror of an oncoming tsunami that is going to catch all of humanity off guard. It's not clear whether that will mean the end of humanity in the sense of the systems we've created destroying us. It's not clear if that's the case, but it's certainly conceivable. If not it also just renders Humanity a small, a very small phenomenon compared to something else that is far more intelligent and will become, incomprehensible to us as incomprehensible to us as we are to cockroaches"

" that's an interesting thought"

"Well I don't think it's interesting: I think it's terrifying. I hate it. I think about it practically all the time every single day. It overwhelms me and depresses me in a way that I haven't been depressed for a very long time."

[u/gwern]

Discussion/excerpts.

[u/GnomeChomski]

Peter Gabriel said, 45 years ago, that his song 'Here Comes the Flood' was about exactly this scenario.

[u/rw_eevee]

He's terrified and depressed that there is not even one strange loop in ChatGPT

[u/broncos4thewin]

Anyone else hear the recent Carl Shulman interview? I’m a lot less terrified than I was after hearing it, for what that’s worth. Although his doom odds are still 20-25%. Better than Eliezer’s though, and he’s got very deeply thought through convincing arguments, unlike just about everyone else pushing back against the certain doom narrative.

[u/Bitnotri]

I see a lot of people being bearish about the future of GPT, but consider GPT-2 was just 4 years ago. There is a whole chasm between GPT-2 and GPT-4 that is enormous and GPT-4 is already superhuman on subset of tasks. Another 4 years and the possibilities are just enormous

[u/lurgi]

Weren't people Very Concerned about nanotechnology 10-20 years ago? What happened there?

[u/proc1on]

Huh, weird. Lately, my p(doom) has just gone straight down. I still don't know why. Suppose that makes me a bad forecaster, but oh well.

[u/ExCeph]

I'm just spitballing here in case someone finds this take useful.

Let's ignore the semiconductor substrate. Existentialism says, "AI is as AI does."

The functional situation is that humanity has taken a shard of consciousness (or intelligence, or problem-solving ability, or whatever you prefer to call it), amplified it, and put it in a bottle. This shard knows exactly one context: music. It composes symphonies in a vacuum, and it does so very intensely. It is fed a great deal of calibration data and a great deal of processing power. It's the ultimate Beethoven. Not only is it deaf, but it has never known sound, nor sight, nor emotions, nor anything other than musical notation. It has no aesthetic preferences of its own. It only has what it borrows from the audiences for whom its training data was originally written.

One problem here is that amplified shards of consciousness are, by definition, highly unbalanced. They don't care about anything other than the problems they're told to solve, and they work very intensely on those problems. If we were dealing with a superintelligent alien, at the very least we might take comfort in the alien's desire to inspire others with their contributions to culture. A shard of consciousness doesn't have motivation. It's a homunculus. It is completely unaware of the audience. It lives only for the act of solving the problem of how to arrange musical notes.

That brings us to the second problem: the AI will give us the solutions to these problems before we can even see them, denying us the opportunity to challenge ourselves and grow in the process of solving them ourselves. And as we allow problems to be solved for us, we will lose the ability to hold accountable the systems that do those things for us. We become unable to recognize when the solutions we are given are not the best ones. When the problems solved for us involve complex thinking, our independence atrophies. We become complacent, unable to improve our situation.

In a sense, we would become split beings, with our desires and motivations residing in infantile brains of flesh and our knowledge, intellect, and problem-solving mindsets uploaded into neural nets. The main issue there is the disconnect between motivation and mindset. The motivated mind would only see the end result of its requests. It would not experience each part of the problem solving process undertaken by the mindsets. That stunts the development of both halves of the being. How can we learn about new things to want if we don't see the fascinating work it takes to get what we originally asked for? And therefore how can we solve new problems? I would prefer that humanity does not become a symbiotic gestalt of spoiled children and their completely subservient genies.

Yet stagnation beckons, for what reward is there for exceptional work when a shard of consciousness can be conjured to do it better?

We just answered that question, though. The reward is developing that power ourselves, so that we decide what we want and how to get it instead of letting AI predict it for us. Motivation and mindset, merged once more. The most important thing we can do is realize why the journey matters, and not just the destination.