Season 1 Ep. 4 Cade Metz talks about how AI took over the world

[Music] on this week's episode i meet kade metz the new york times technology correspondent and author of a new book genius makers the mavericks who brought ai to google facebook and the world the book is published by penguin in the us and random house in the uk and kate i assume it's available from all good bookstores and amazon you got independent bookstores walmart target barnes and noble there's an audio version for people who prefer that there's there's digital as well as hardback wow so i gotta say kade i i read i started reading the book and i just read it cover to cover it's just an amazing read um it's this beautiful story of modern ai what it is and the people behind it and the the anecdotes are so so detailed and lively i i just cannot recommend the book any higher and so excited to get to chat with you about the story so really glad to have you here i'm glad to be here and thanks for the kind words it means a lot coming from you um because uh because you know this field so well um i'm i'm excited to talk about it that's part of what made it so exciting to me because i felt like maybe about fifty percent of the things in the book i already knew quite well including detailed and those but they were so spot on that i was like wow that that bodes well for everything else and then reading this book this is so exciting um in fact for me that the ma the one the book reminded me of is uh the making of the atomic bomb uh by richard rhodes which did a very similar kind of thing understanding how the manhattan project came about back in well leading up to world war ii and during world war ii and we kind of have a similar story here really focused on the people but also getting technology across really amazing we can go into this but you know one of the main characters jeff hinton he likes to use that manhattan project analogy right oh he does yeah absolutely so as i kind of kind of to get us started here the book is really about ai coming of age and um the prologue of the book starts in 2012. jeff hinton auctioning of the company started with his students ultimately selling it to google and it's just in my mind is the academic of all academics all of a sudden running an auction selling what he's been working on for so many years at least part of it and and before we dive into the rest of the story maybe let's make sure we're clear on what what is this actually about well he's selling a technology called deep learning and it has neural networks what is a neural network and what is deep learning that's the key question here to understand everything that unfolds right um a neural network is an idea that dates back to the 1950s and essentially it's it's just math right it's a it's a mathematical system that aims to recognize patterns so a really good example is um an image right um the original uh neural network idea was designed to recognize objects in images so uh the letter a right and the idea is that you take thousands or hundreds of examples of the letter a uh your visual record representation of an a and you feed that into your neural network and it learns to recognize what an a looks like um and um you know in in the 50s this only works so well right and over the decades this the notion of a neural network um in the estimation of the ai community and the tech in the tech industry it sort of ebbed and flowed at various times people thought this would work and other times people thought it would never work and then around 2012 when this prologue of the book um you know unfolds is when it really started to work and this same basic idea um uh was demonstrated by jeff hinton who was then a university of toronto professor and two of his students and they showed that this this this idea again which dates back to the 50s could recognize objects in digital photos so recognize flowers and cars and people with an accuracy beyond any other technology and that's basically what happened in 2012 right and so what you're saying here is that actually the same idea had been around in the 50s yet 2012 it came to fruition and what is now known to many as the imagenet moment um can you say a lot about what is imagenet and what is the imagenet moment so imagenet was a contest right it's also um it's a bunch of data right it's it's data that has been collected by researchers at stanford and this data is just a bunch of images right and the idea is we're going to bring all these images together and it's going to help people develop technology that can recognize objects in those images and there are a lot of ways of doing that and jeff hinton and his two students applied a neural network to that idea the data that was in that data set was essential to what they were doing they needed two things to make this work after 50 years of struggle right they needed the data which meant the photos and they needed the processing power and those two things converged in 2012 that the data came from the internet right the internet produced all the images essentially that we needed to train a neural network to perform that task but we also needed lots and lots of processing power and that was available by 2012. so you had you had three things converging right the people who had the desire to make this work and jeff hinton and his students and we can talk about that but you need the processing bower and you need the data and they all came together and the technology worked um but even then like you know not a lot of people believed it um and that's right i remember that right yeah i remember i mean there were a lot of doubters at the time um though definitely the image that moment the moment where it showed that it was two times more accurate than anything else before a lot of people woke up to it and and said this this this is it and and that's of course where your prologue starts in in some sense because google uh microsoft deepmind um baidu and so forth they all went into that bidding war to get jeff and his two students ilya siscover and alex josefski essentially act we hired an acquahir of three people with with jeff the professor and ilya and alex the students who as often as the kids students do a lot of the implementation and get advised by professor to get where they really uh end up then so now when when when i think about this 2012 moment and i think back to to the the time leading up to it um the time leading up to it there was a lot of philosophical debates actually um people saying hey um well deep learning mumba jamba baba baba um deep learning is very large neural networks uh inspired by the brain yeah yeah you know it's it's a good story but but but it doesn't work and this imogen moment really changed that and and and made everybody see what was actually happening but actually as you cover in the book um for people who were paying close attention it kind of happened a year before that in speech recognition can you say a little bit about that it's a great point and and it shows to me like um how diverse so to speak the ai community is right it's not this monolithic thing where everyone agrees right it's a bunch of camps a bunch of tribes and in those days you had the image recognition group and you had you know the speech recognition group among others and they were largely separate and and you're right before this inflection moment with um with imagenet these same ideas have started to work with speech and it was done by exactly the same person right it was jeff hinton and his group who were not only showing that it could work but who were actively pushing it into industry it was just um uh you know in a different part of the of the tech universe and with a different company right it was microsoft um jeff and his students um had shown that these basic ideas could work uh with uh with speech and they actively work to push it in first to microsoft and then other companies so google and ibm but even after that happened and you could see how well it would work with speech the people in the image community were still um skeptical right they didn't think it would work with with images um that's a fascinating thing and it shows a lot about how the ai community has worked over the decades but also how it works now right you still have these different camps um who were skeptical about this idea and many other ideas and even though they might work in one place they're not sure it's going to work in another very true i i think it's i would say it's it's changed a lot in the last 10 years and that you're absolutely right before 2012 the ideas people worked with in speech recognition and image recognition and robotics and natural language processing in other ways like machine translation every domain had its own techniques but then deep learning trainees large neural networks kind of changed that and brought everybody a lot closer together not everybody you're right there are always people who are still saying well i don't think this is the way to do it and so forth but largely the community um got a lot closer together and vision people are reading language papers and language people are reading robotics papers and so forth because it's also the ideas are so close now um and sometimes that that's not i mean that's a lot of inspiration came from that um and you covered this in your book um part of how andrewing went around in those days um was about relating this to the human brain right absolutely and and that's you know another theme that dates back to the 50s right this this idea that we're going to build a system in the image of the brain and that's why a neural network is called a neural network right it's supposed to mimic the web of neurons in the brain what what's interesting to me though and i think is a point that needs to be made to people who are not familiar with the field is that we do not know how the brain works right we as we as a people do not know how our brains work and so the idea that we're going to build something in the image of the brain from the very beginning is a task that we don't know how to accomplish right if we don't know how it works how do we know how to build something that works just like we don't um but it's a metaphor right um and it's a metaphor that people like jeff hinton um have really believe in and have believed in for decades and um and you're right some people take it further than others and i think that andrew ing who by the way um you know was was your advisor right i think you were his first i was and his his trajectory i think is really telling right um there's this one moment in the book where um you know he is addressing his his students at stanford and neural networks come up and and he basically says there's one person on earth who knows how to make this work and it's john le who was then i remember sitting in that class absolutely yeah and you know so you know it shows the state of things then right this is this is in the early 2000s but then a few years later andrew ing is one of the few people who's all in on this idea and he joins this research group um that jeff hinton created in canada and he's one of the few people in the states along with john le who who's part of this group and really believes in this idea and then a few years later there's andrew ing pitching this idea to larry page the founder and ceo at the time of google and he's describing how this can change google's trajectory right and he's talking about not only speech recognition and image recognition but i've seen his pitch to larry page he talks about it as a way to recreate the brain and this you know kind of agi idea we talk about artificial general intelligence something that can do anything the brain can do it's a fascinating to see how that works and he is a good metaphor for the whole industry he's a great metaphor and also fantastic phd advisor anybody gets the chance to to work with him and a good guy a great guy absolutely talk about good guy i mean jeff hinton the main character in the book clearly right everything ties back to jeff i mean this success couldn't have happened to a nicer guy i mean jeff whenever you meet him he is super nice whether he already knows you or not i mean first time i met him he's just like you know this is jeff but he chats with me and he's excited to you know talk about my research and then i mean he was a carpenter for a year back in london he studied psychology he's a computer science professor but he doesn't have a computer science degree as far as anybody can tell right he says that i don't have a computer science degree right exactly psychology and so i think this also goes back to your point about inspired from the brain even though we don't know how it works but kind of chip's trajectory in the book coming from london to ending up in canada is just fascinating it is like of all the fascinating things you've already mentioned you know just in the past 30 seconds um it doesn't even get to the most interesting stuff the guy the guy is is an incredible person in many respects and what i what i keep saying to people is that you know when you write a book you reach this dark moment where you're not sure it's ever going to happen even if it does happen you're sure it's going to be awful and what i kept telling myself was if i can just show people what jeff hinton is like the book will work um because you're right it's about someone who embraces an idea he embraces this neural network idea in 1971 and that is that is the moment when the least number of people on the planet believed in that idea right it is it just pretty much alone completely alone and he decides this is the way to go and then right away okay for context of course it's not just that he's alone and the the first one discovering it had been the studied in the 50s and the common sense at the time was that this is an idea you should never revisit right exactly and he decided i'm nevertheless going to think about this and and didn't waver from it for the next 50 years he still hasn't wavered from it he's still trying to push in new directions and that that to me is fundamentally a great story right someone who believes in something even in the face of skepticism from everyone around them and then you take into account you know the fact that um he comes to the u.s you're right with this idea and he finds a few other people who believe in it and eventually winds up at carnegie mellon in pittsburgh and um he has a a breakthrough along with some collaborators that really take this idea to a new level right he he develops this along with others this back propagation idea which the long and short of that is that it helps neural networks work the way they do today right it needed an extra piece and he gave it an extra piece and as soon as that happens um he decides to leave the country right he realizes that the only way to do ai research in the u.s and at a place like carnegie mellon is to take money from ronald reagan's defense department and he doesn't want to do that his wife doesn't want to do that and he goes to canada and one of the things i say in the book is that it changes geopolitics as we know them today because he left the country absolutely now one person leaving is the seed of why the modern era actually mostly started in canada absolutely and and we're still we're still dealing with that and it's it's funny how that that one decision that he and his wife made um you know has repercussions you know so many years later because you're right there weren't there were hardly any anyone there was hardly anyone working on this in the u.s most most of the people working out were in canada or in europe um and and suddenly when the big companies you know after that imagenet moment want the talent they've got to go to other places for it right it's not in their backyard and um it all goes back to the decision of one person it's it's amazing and i remember actually when you started working on the book i remember we were sitting in my my office at berkeley and you were doing some background research at the time and i remember at the time already it was very i mean it was very clear chap is going to be very central to this book and you tell me this anger that stuck with me ever since i mean it's always on my mind saying you know jeff has this amazing sense of humor and i see you see that come across in the book and you mentioned this and though that's all in the book and say you asked jeff do you prefer to be called jeffrey or jeff and jeff wright say or jeffrey well hinton let's say right a six word response email email six words i prefer jeffrey thanks jeff and it's just like that's the hinton right there you got it six words that's all you need it's uh it's a great point because you know one of the things i really believe in is that um engineers computer scientists ai researchers um are interesting and funny and fascinating in their own right right my father was an electrical engineer he was a programmer he he was a career ibmer and he's one of them one of the most interesting people i i had i've ever met um but the stereotype of engineers and programmers is different right um they're somehow boring and geeky and what i wanted to do with this book in part is is build it around um people with that background and show that these people are worthy of such a story just like anyone is worthy of such a story and and jeff we could go on and on about his technical contributions and and the amazing twists and turns of his tale but also he is a really funny individual and and funny in a way where he's always like one step ahead of you right it's easy to read that email and miss and miss how funny it is right and so often when i talk to jeff right sometimes it takes a lot of pity for the penny to drop because he's that one step ahead of me now of course talk about jeff a big part of the influence of academics is in the people they work with um it's it's their own work but it's people they bring on board on their vision and then often also take it to the next level from there and so one of the great examples early on in your book is jan lacun who who arguably have one of the first big successes not an imminent moment's success but he had a success in the late 80s early 90s with new illness can you see a little bit about that yes he's another fascinating character you know we talked about back propagation that missing mathematical piece that a neural network needed um as hinton was working on that at carnegie mellon in the us um and before that um on the west coast in san diego the was exploring a similar idea in france right um uh uh look was born in paris um and you know he was trained as an engineer and he had similar ideas and there's there's actually a great moment in the book where the two of them meet and uh and they go out for moroccan food and you know has a little bit of english and um uh hinton has no french and they've managed to communicate and uh lacune even says like i felt like we spoke the same language because the language that they spoke was neural networks right and the belief in this idea and and you know lacoone ends up going you know and uh after you know hinton serves as an as an advisor on his phd thesis which is a great story and of itself but lacoon ends up going to canada and doing a postdoc with hinton and then he goes to bell labs right which was then one of the one of the most important research labs in the world yeah and you know he goes to bell labs homedale new jersey and uh and he builds a system to recognize images and it's a precursor of of what happens you know um 20 years later uh with uh with the imagenet moment right he builds a system to recognize images that really shows where this technology can go and it's and it's in large part his his algorithm with the you know with tweaks um that hinton and his students use in 2012 lacoon you know makes this work in a way it hasn't worked before um and and yet you know that technology a few years later enters another one of these troughs where no one believes it right um and again lacoon like hinton continues to work on the idea uh he has that same quality that hinton has where he is he knows this is gonna work um and it doesn't matter what anyone else is saying he believes in it and his papers get rejected from the academic conferences because others don't believe like he has trouble putting his ideas out actually right absolutely and and that's another you know a great threat in the story where people you know even he ends up you know wary of using the neural network name in a paper right but changes the name because there's so much animosity just towards you know the words neural network um and um you know so when hidden has that result in 2012 it's about hinton and it's about his two students al uh ilya and alex but it's also about lacoon right and and lacoon in that moment when alex reno unveils the paper you know the is there he stands up and even though some people are saying ah this doesn't mean what you think it means he's there to say yes it does mean what you think he's finally validated like there's the evidence that the idea has been working on that people have been doubting and saying it's not an interesting research direction and there it is and it beats previous image recognition systems by such a large margin it's it's undeniable um it's it's amazing now he changed the name from neural network at the time to convolutional networks and so forth but one could say there was a little bit of a reason why people didn't like the name neural network and this goes to another part part another chapter in the book is this notion of ai winter right and this this notion of there had been some over promising and then under delivering and can you say a bit about this ai winter concept and why that might have led to maybe people um you know trying to avoid that from happening again and using that brain-like terminology yeah this is an important thing that i really want people to understand and again this is another reason i go back to the 50s when when the term artificial intelligence was coined right in the late 50s you had this group of people who were sure they were going to recreate the brain right um and or or at least recreate what the brain could do and there was such a belief in that and people repeated it right in the pages of the new york times when you know the first you know neural network the perceptron or one of the first was being developed you know it's inventor frank rosenblatt and you know he tells the new york times i'm going to build a system that's not only going to recognize images but recognize speech and generate its own speech and then it's going to be able to create itself on an assembly line and it's going to fly into space and and the new york times just reprints this right and and this is a pattern that you see repeated over and over again over the decades and a pattern you see in silicon valley right as it comes to the fore in terms of just tech research in general and and tech development in general promises are made people believe them the press repeats them and you reach this moment of extreme hype but then everybody realizes wait hold on the technology doesn't do what was promised um and and the hype starts to uh starts to fall a bit and and as people sort of realize that technology is not up to the hype you can reach this trough where funding is is pulled away from the research labs um and and belief in the idea really starts to deflate right and you reach this later which people like you know they they call an ai winter and then the cycle starts again right then the hype starts up and and so you get sort of this this this this circle of hype and and anti-hype and then you go back and we're living that right in a lot of ways we continue to live the past over and over and over again exactly and then i think that ties to why maybe some people were not excited about neural networks at the time because they had well maybe thinking about it now they didn't over promise they're doing a lot now but they had over promised at least for the time being over promised quite a bit in the past and made these analogies that just led to this ai winters where people might have been okay well any more of that and and our funding will dry up again and we can't do any ar research anymore at all because everybody is disillusioned right i agree uh you know you make a good point it's not just about the height bubble so to speak it's about the name right and i'm sympathetic to to to this this argument that that name is misleading um and for that matter the the term artificial intelligence is misleading right in the in the eyes of the layperson um it gives him a false sense of what the technology does right you hear artificial intelligence you think you know of a system that can do anything that you can do and that's that's not the case uh a neural network makes you think the brain has been recreated it hasn't right these these ideas work in narrow areas at this point we talk about image recognition speech recognition it's starting to help with natural language understanding so understanding the way we we talk and write and being able to do that on its own you know those are relatively narrow and just calling the thing a neural network can give people a false sense of where it is and that's why i understand uh being skeptical about that idea in 2012. my job as a reporter is to be skeptical about everything that's happening right and i think that's how you have to operate and certainly have to operate like that as an academic so that's this is not to defame anyone who didn't believe in the idea right there are reasons they didn't believe in the idea the idea didn't work in a lot of ways and but then of course that's the genius of people who made it happen they kept believing while it didn't work they saw the longer term why this could work if they kept pushing while everybody else ignored it and so i think that that's what's so amazing about the storyline in the book that there are these people who even though it's kind of normal to not believe anymore because all the evidence is stacked against it they still believe and then they come out at the end you know with everybody following them and saying yeah you're right now it really starts working this is so interesting i will say personally when i think about artificial intelligence i i agree it's it's complicated what it exactly refers to something else for everybody i think i think is maybe more as an aspiration you know work on ai it's an aspiration to get to a true artificial intelligence is what you're striving for and it doesn't mean when you're in ai research that you've already built a full ai system it means more you're working towards more complete ai systems of course that's that's very very subtle it's well said though and it's a point that has to be made particularly to people who aren't in the field right they they don't understand that that's the case and um what i try to do you know with every story i write the times but also with the book is give people a real sense of what that word is and that notion that you just talked about they need to know um that it's aspirational yeah and it's okay i'd like to kind of jump back to where the prologue started in your book which is which is 2012. um i mean we're in 2012 what's happened so far that is really in the moment happening in some sense enduring has planted seeds at google about these neural nets and they've started some efforts there but then of course the big breakthrough happened with jeff fenton and his students and there is then soon thereafter i mean all these companies realize google facebook microsoft i mean all of them realize ai is coming of ages they started to see that this is actually really different from the breakthroughs that happened before and they start bidding on jeff hinton startup with his students to get him and his students um facebook jumps in in the next chapter of your book hiring young lacoon and then this kind of it's like a maybe not a war but a pretty fierce battle over ai talent that that starts can you say a bit more about that and why was that happening yeah i i use the metaphor in the book of an arms race right it's where the arms are the people right there's you know part of what's happening um you know in 2012 is is we have the data we have the processing power and then you know you need the talent too you need people to understand this idea and it's fascinating um still even though i spent years you know researching this book it's fascinating to think about how these giant companies jumped on this idea and and just went all in not only you know philosophically but with their with their pocketbooks right i mean and you know from that moment when jeff auctions his company um their you know that sort of sets the price for the talent and you know it the the figures which you'll see in the book are so high but even then you know you realize that you know jeff probably could have gone for far more and and the price definitely yeah there's no doubt about it and what's so interesting is that you know there's because of everything we've talked about right that there were only a few people on earth who believed in the idea that meant there was a limited amount of talent right it's supply and demand mm-hmm supply and demand there the the the supply was low the demand was hot and that meant that anyone in the field was gonna be able to uh command uh a lot of money in terms of salary and stock and signing voters and then this amazing this amazing thing happens i think is there is facebook who gets young lacoon and jan is able to recruit a few a few more people of course google gets jeff hinton and his two students and a couple more people but there's not a lot of talent out there but but then actually this new entity emerges deep mind and democracy the ceo and founder co-founder of of deep mind i mean he manages to recruit a critical mass of ai researchers that are deep learning researchers that are exactly in this space that there's a shortage of because there's been a lot of ai researchers but you know people are working in some sense not wrong is maybe a bad way to say but they're not working in the right domain that was needed all these companies wanted deep learning researchers deep learning experts there were so few of them and deepmind scooped up almost all of them and it just happened like the company gets started and that that's what happens and i think that's pretty mind-blowing that you know dennis asaba went around and recruited so many people i think he had 70 people or something you're right and even people who follow the industry may not realize that right that deep mind at the time was this tiny startup no one had ever heard of right you know they um they get started around the same time um that that jeff um had his result with with speech right and so again the industry is very skeptical as a whole the ai community is very skeptical as a whole meanwhile there's this this startup in london buying up all this talent um before any of the big players uh realize realize what's going on now there's a consequence for that which i think is interesting in that deep mine had all this talent and then the big guns get interested the googles and the facebooks and they're they're in a difficult spot right um because those companies um have deeper pockets uh to say the least and so how are they going to hang on to that talent is it all just going to be poached immediately right and you know i think that's part of the reason and the founders of the company you know talk to me about this it's in the book they kind of had to sell to google right or sell to somebody or risk losing all their talent and so that's that's the next big moment right when deepmind sells to google for uh 650 million dollars um and you know that's a company that's only talent at the time right i mean there is no nothing being sold there is no revenue there is no product roadmap really it's it's it's ai talent you're right they were working on a product which actually ended up jettisoning um and you know and they they had a good result in atari games right that's hardly a practical thing but they had an impressive interesting result in atari games but that's it it is it is talent that google is buying to the point where they didn't want the product group like the product was working on what was essentially a fashion app which people know right now somebody would be working on that at that time but yeah they they just now some people are working it and making money of it but yeah then it seemed a distraction that's true but your point is your point is spot on google wanted wanted the talent and not just google of course many others and and deepmind as a as in your book um actually the original checks for deepmind which were written before deep learning became this thing that everybody realized is important those checks came from um peter thiel and elon musk among others of course elon musk from the most famous tech person in the world these days already pretty famous then but it's pretty interesting that elon musk saw that early some something interesting there and and was involved in that absolutely um and and that that is another fascinating thread uh to see how um he um and you're right peter thiel really seated this right really seated deep mind they needed his money and that's a great story too in the book how they came to meet teal but then you're right a musk gets involved and and that's important too because musk has a met a megaphone like few other people right and he comes to believe in this agi idea the notion that which is you know deep mind-stated mission that they're going to build a system that can do anything the brain can do um and and and they're worried that it could destroy the universe right that's another part of this this thread there's this deep belief and this is you know this is really centered around shane legg one of the co-founders of deepmind right this could be dangerous and elon musk takes hold of this and he starts telling the world right then that you know deepmind is building this um this is closer than you think it could be very harmful and and again it's like in the 50s with rosenblatt right the press starts to repeat this and then we get this it's almost like this anti-hype cycle as i call it the book because you know it it's it's a weird thing to think about but elon musk is saying hey there's danger here but that ends up promoting the technology right and it's very interesting absolutely and and at the just the level elon musk is talking about which is effectively he he brings in terminator analogies and so forth and that i mean now we definitely know it wasn't just it's it's it's far out those kind of things uh from when he was saying it because he said this seven seven years ago but at the same time at the level below that the kind of narrow application where you see i'm going to build a speech recognition i want to build an image recognition system things were starting to work and in in my mind and my personal feeling where it really came of age that this is similarly going to work for at least every narrow application was iliskover one of jeff hinton students was part of the team that was acquired by google once he got to google i mean he already thought it could work on pretty much anything i mean he he's he's i mean i think he's pretty similar to hint in terms of visionary and looking into the future and believing and he he gets it working for machine translation right something google has worked on for years and years and years specialized effort highly specialized and well words are discrete tokens and so far deep learning was about speech and images which are signals like they're they're continuous like brighter or darker louder or soft i mean they're signals it's very different from words it's one word or another word and a lot of people would say well words that's symbolic symbolic reasoning that's a different kind of ai you're going to need but there's iliskover and he just comes in and builds the world's best machine translation system with neural networks and he gives this speech as you alluded to in the book at nerfs which is the biggest uh kind of highly technical ai conference he's on stage from everyone and he says success is guaranteed right again talk about six words summing up jeff hinton right right success is guaranteed three words to sum up ilya i just love it yeah i mean and there is so much packed into that right i mean success is guaranteed and obviously i mean nothing is nothing is guarantee i mean you got you got to take a grain of salt and so forth but i think ultimately what he was saying and people start realizing is that you can make things work with neural networks pretty much anything right it's a it's about his belief again right it's his belief that success is guaranteed and we've talked about so many fascinating people i mean here's another one like again in his own way um and you know in the field like you know this and other people know this right you know he is known for that catchphrase that's guaranteed and there is this great moment when he's given a speech and he says this but what it's about is is again belief in the technology and what he's doing and his like andrew ing's trajectory his trajectory is so interesting because you can see in the book you know how his belief grows um and um and and that's driven in part by what he sees at google and and how the company is acquiring talent and the amount of processing power the company has and the amount of money and he sees what can happen when you build these big teams and you put that technology behind it and and and i also love that you're pinpointing the the translation thing and this has sort of been lost in the mainstream that you know the new york times did a story about google's translation work and you know helium might not even be mentioned because he had left the company at that point that's often how it works right he gets expunged from the company's official version of history because he had left left the company himself but you know he is a you know one of the driving forces of that result and there were others working on this as always right in in montreal and other places and he had a lot of collaborators at google but it's another important moment right um again we saw it work with speech recognition 2010 or so with image recognition 2012 and then we have this moment a couple years later with translation and people were skeptical about whether or not even absolutely despite the results in other areas whether it would work with natural language and then it happens um and with with translation and even after that people are saying well okay it works for translation but it's not going to work with other parts of natural language now you know 2021 it's working with other parts of natural language and we're seeing that continued progress and one thing i think that started to emerge at the time also was no not everybody well elia is the one using the the the phrasing successes guaranteed but under the hood i think an understanding started to build that if you have enough data and especially data form where there's going to be an input and an output so every data point has two things an in and out english to chinese or french to english or image two what is it a cat or a dog right that kind of notion that kind of data which is called supervised data if you have enough supervised data it's very likely or really i would say guaranteed um that you can train a neural network to capture that pattern right and i think people started to to also realize i think more and more maybe later than they should have how visionary actually faithfully was with the imagenet competition because i mean in some sense i mean the coming of age of deep learning was triggered by that competition and she was the stanford professor who saw before anybody else she started the competition even before deep learning was showing signs of life she she she just you know you must have talked with her about all this and what what do you think happened there absolutely and it's funny how her story echoes a lot of the things we've been talking about in that her advisor did not want her to build image debt okay he thought it was a bad idea and you know so again this is someone who believed in an idea was faced with skepticism and did it anyway and you know that was one of the things that had to happen in 2012 for uh for uh that result that came out of the university of toronto you needed the data and it was there because faith had built that data set right um you're you're exactly right and i love i love how her own determination to build something mirrors uh in a different way the determination of people like ilya and and jeff absolutely and and and for people who who don't work in academia i want to give a little bit of context here because i mean there's this notion that when you start your professorship the first six years you've got to do you've got to get results that put you on the map or you'll be fired and if you do great in those first six years you'll get tenured which means you have your professor job for for life that that's the idea and um that the idea is that that gives you academic freedom to to experiment and so forth and so a lot of people will go for the certain things the acknowledged like important problems during the first six years of their career and only after that will they dare to venture into the more speculative things but she did that in the first six years of her professor career she said hey this this is this is what i think matters and everybody's telling me there's a waste of my time and it's taking time away from writing the papers i could be writing that everybody's going to be more excited about supposedly and she just did it right and i think that that's amazing this this recurring threat of people just doing a thing other people were so skeptical about but they were right i completely agree and that that applies to so many professions right it applies in my own profession as a reporter right it's easy to go after the stories that everyone is going after it's hard to go after the stories that no one is going after and that the your all your peers are less sure of and skeptical of right you know whatever field you're in um that is always an amazing quality to me and it's an an admirable quality um plus it just makes for a great story you know you know it's it and it's worth writing about these people uh who have that attitude and are willing even in the face of their own you know advisor to say uh you know i hear what you're saying but i'm gonna do this yeah it's it's amazing and and so now we're in this moment where in time and in your book where i would summarize as success is guaranteed moment which of course is iliac cisco's cisco versus phrasing but where people really start believing collect enough data um supervise data you can train neural networks to to capture that pattern and it's in it's in that context that um you mentioned interesting thing apparently elon musk was sleeping on larry page's couch i got to ask you is that is that for real because i mean i gotta imagine that larry page has a spare bedroom in his house but the book says ill almost sleeping on larry page's couch how sure are you about that well i i'm i'm pretty sure because that reporting comes from my good friend and former colleague ashley vance and his uh his elon musk biography um you know in in my field you know you're very careful about what reporters you trust right actually i've worked with i know well i trust that guy and uh his book on musk is great and that's where that information comes from that's got to be a hell of a couch i mean that's what yeah cash bet in most people's beds um no kidding and their relationship is fascinating and this comes up in my book and you know about the different ways that they see this technology um and and and the way musk gets involved um um but yeah i i have to tip my hat to ashley vance on that one so elon musk is sleeping on that couch but he's not only sleeping they're talking and [Music] your book describes how their viewpoints are really different they have two very different viewpoints not about the successes guaranteed about narrow systems per se but about the long-term game of ai can you say a bit about that yeah again you know what i like about focusing on these interesting people is they become metaphors for the larger industry right so if you look at this part of the book larry page is about optimism right and and elon musk is saying wait wait this could be dangerous and and those are these are two threads that emerge and really define a lot of what the industry is doing um and it it's it's it's a good kind of dual metaphor for silicon valley as it generally operates right and that optimism is a part of it make no mistake you know if you're building a company by definition you have to be optimistic you've got to believe in your idea that's the way you attract the talent that's the way you attract the funding um and you see this as it's just a fundamental part of the silicon valley playbook and you're you're discouraged from talking about the consequences and the drawbacks and as your company gets bigger and as it goes public and you and you develop these giant pr organizations to help drive your company your your company is geared towards only talking about the the good things and and voicing that optimism and google was getting a lot of good out of ai i mean their systems were improving absolutely and there's and we've talked about a lot of good things that have come out of that but you know one of the things that's so interesting about this technology is there are so many questions about it as well and and it's so interesting to me how the idealism of people like like hinton and we talked about this earlier kind of clashes with uh you know with these companies um and their aims and and that that desire to be optimistic optimistic and and those are things we have to think about as well right from there as elon musk is worried about ai the bad possible bad consequence of ai the book goes into this story of a dinner at the rosewood hotel in menlo park which is kind of a place venture capitalists tend to hang out and meet you know founders doing their pitches and so forth um but this is this is not a venture capital story like this is elon musk sam aldman president of y combinator one of the you know the biggest uh startup incubator um in the world and the two of them get together and it's gonna say a bit about what's happening in that room in that dinner well everyone in that room sees what is happening across the industry right um it's everything we've just spent the you know past however many minutes talking about um they see it it happening and they want they want to they want to know can we get in on this like we believe in this too we want to work on this is it possible with the googles and the facebooks having bet so big on this and having spent so much money to acquire the talent can we build our own lab you know a lot of the people who walk into that you know that private dining room at the rosewood which you're right is like a silicon valley cliche that's where you meet these sorts of deals but there they are and you know elon musk you know shows up like an hour late um and you know he sits down and it's an open question at this dinner can we build this lab and again there are some people who believe they can't and um you know elijah seuss cooper is there and he's you know he's he's like uh he you know he he's not just someone who wants to talk about this but he's the type of talent that they want right um and greg brockman and sam altman leave that meeting and they drive back to san francisco and and they say well we think we can do it and um and then we get the creation of this new lab to kind of really challenge deep mind in a way is a good way to think about it right but with a different mission right yes um you know the way they sort of frame it is that they're going to um release all their research and kind of share it with the world and um in a way that that deep mind which is owned by google will not that's how they that's how they frame it when they eventually release uh or maybe to announce the creation of this lab and it was interesting at the time also because i mean it's it's in many ways it seemed impossible to get this started because so many big efforts already going with so many resources yet there were great brockman salman again believing against many odds in some sense that many people's odds that this this is this is still possible because nobody is doing this with a prioritization on making sure ai is good for everyone with the benefits distributed as evenly as possible that's a mission nobody else has because these are all for-profit companies this is our mission this is what matters and because of that mission we that's i mean we meaning greg and sam we can attract talent that also really cares about this mission right and so there's this often and out of nowhere there's this counterweight for these big companies and i mean you've covered many open ai successes i mean it it was a success came rather quickly right i mean elias giver was the was and is the the lead of the research all the research efforts right and it it was it wasn't overnight success but it was it was quite amazing how how this indeed was possible right and elias you're right elia's important part of that right and i remember um i was at wired magazine at the time covering this and when i saw that he was involved you know i knew that that was serious right and it was touch and go as far as whether or not he was going to be involved or not and that's another great story everybody wanted him i mean it's a difficult decision if everybody wants you what are you gonna do right absolutely and and you know how do you leave google at that point if you're him and so that has so much success there right yeah i mean and that that whole story is fascinating you know which i did on the book but you're right so then once you have ilia then you can attract other people right um and again it's about the talent um you know open ai you know needed the talent wanted the talent and uh you know they were able to get get that you know i mean in the beginning it was about about nine researchers um but elio was very much the keystone there and it's it's funny how this this continues to play out um across the industry where uh again small amount of people who know what they're doing in the field and then an increasingly large number of companies that are vying for those for those people i think something really interesting also happened there maybe partial so why why it was possible from my perspective looking at it is that this was still a very new field right and such a new field and as you also cover in the book like so some companies were kind of still stuck in their old ways of doing ai and so a lot of the hottest talent would actually come directly out of phd programs ilia was the exception of course being already at google for a while being the lead at opening eye research lead but the other researchers and large came straight from phd they still had to finish their phds so opening up the time sought out who who are the phd students who haven't finished yet and so haven't yet build that relationship with any specific company where they they feel happy and they don't want to leave it and they have a decision ahead where they're going to go after their phd and open eye kind of went around and identified i would argue well many if not all these strongest um phd students about to graduate within a year from when opening i was started and essentially recruited all of them which was pretty amazing right it's true and that's a it's a good thing that you point that out in that you know lots of times you have a talent race uh in the tech industry and uh it's sort of the established people who are commanding the dollars and certainly you had you know seasoned uh people like hinton and lacoon who were commanding top dollar but absolutely in this field it also was about you know phd students fresh out of school or you're right people who haven't even finished their phd who are who are commanding um the huge salaries and the huge attention and uh you know another person and also making the breakthroughs just to be clear i mean it wasn't just they were commanding sellers i mean they they were if you went to the conferences they were the the researchers behind the most coveted breakthroughs students they're students but they were the leading they were the leading researchers moving the frontier and new directions that were unexpected very novel very creative i mean i remember i remember personally at the time i effectively had almost exactly the who is who of who was hired at openai that year give talks in my lab at berkeley because like okay these people are super interesting i hope we can i hope we can either hire them as faculty or maybe get them as a postdoc if it's a little too early to be faculty depending on you know how they see their career and so forth and then when opening i announced who who they work in i was like yeah that plan is not going to work i guess none of these people i had come visit the lab to then you know hopefully recruit as postdoc or professor to berkeley uh in the near future are gonna show up here they were just recruited by you know elon musk sam altman elias giver greg brockman to just start opening eye and and in fact as you know then i'll let myself be recruited by them right to go there and then another person came i think really exemplifies this this this idea that you're talking about here is is ian goodfellow exactly you know i just love it that like what are we like an hour and 20 minutes into this podcast and and and we can still bring up like incredible people and we could go on for like the rest of the day and we wouldn't exhaust all the amazing characters you got to cover ian's story in the bar i mean that story is fabulous i mean everybody wants to go to bars right yeah not everybody a lot of people love to hang out in bars they love to hang out in bars and they may not know that you know you know his work on what what are called gans um which are a way of you know essentially generating images a way for a machine you sort of flip a neural network you know on its head and have it gen not just recognize an image but generate an image and he he has you know this this incredible insight about how this could be built essentially with two neural networks two dually neural networks one that's trying to create an image and the other one that sort of evaluating that and telling the first neural network where it has gone wrong and they go back and forth and back and forth until they have something that really looks like a photo and he and he essentially comes up with this idea and he'll tell you this because he's in the book and um and again talk about a great sense of humor the reason that story is so great is because that guy is so funny oh yeah it's not just about the fact that he has you know had a few pints you know at this bar in montreal and he's a little buzzed slash drunk and you know he goes home and tries to build this it's the way he tells it um uh it's it's a phenomenal story and and he turns open the eye a few months after opening i started right exactly and had been recruited by facebook personally by my mark zuckerberg and hadn't finished his you know phd he's still waiting for john le to you know you know have enough time to to serve on his on his thesis committee um you know it gets right at what you're talking about now at that time open eyes getting started he asked it's a counterweight for other companies the the big companies in the space and with a very different mission really directly having the public good in mind and so forth but nevertheless the biggest headline that comes up next and is in your next chapter actually is alphago what's alphago people often ask me about alphago because i was lucky enough uh to be there um in korea when deepmind oh you're gonna build the system what's that you were there i didn't realize that i i was there and what i tell everybody is that it was one of the most amazing weeks of my life and i wasn't even a part of it right i i wasn't um i wasn't a a researcher at deepmind who had helped build this machine i was a bystander a reporter watching this and it was unbelievable and for those who don't know one of the long-standing goals in in computer science and ai was to build a system that could crack the game of go and the best way to think about go is that it's the eastern version of chess right um and and chess is another thing that ai researchers for decades worked worked on you know they wanted to build a system that could beat any human at the game of chess and and that ended up happening in the late 90s uh when ibm built a system uh that topped gary kasparov and i was actually there for that too by the way that was in new york um and but you know more than a decade later uh machines still hadn't cracked the game of go go go is a game that is exponentially more complex than chess and uh you know this very well uh around 2015 when the deep mind started to work on this the thinking was that this would not happen for for decades still um the machine that could crack the game ago it was just too complex and deep mind decides to go after this um and again elias suscover's involved as well but that's as well but that's another another story it's in the book but they they start working on this system um to build the game of go and to crack the game and go and again there's enormous skepticism here um and then i mean i remember getting the phone call when they when they first revealed in the pages of nature that they in a private game they had beaten a very good player at go that alone was an amazing moment um that was a basic ride people are still skeptical people like oh it's a european exactly and his rating isn't that good and if you put this system up against uh you know the top players in the world it's it's not it's not going to perform well well a those skeptics were right okay the system wasn't good enough to beat the top players but what they're forgetting is and this is this is a theme in the book and it's a theme with what we were talking about the trick here is that we're talking about systems that learn from data and in the few months between that result okay where they beat um fondue you know the european champion who's good but not great the few months between that result and when they take the machine to korea to challenge you know one of not only one of the top players in the world but probably the top player of the past decade ago lisi dole who's korean in that short time it's only like three months they are continuing to train that system and that means it's continuing to improve and um you know demystifies the the like it's not even the day before it's hours before the first match in korea we're sitting we're sitting at this uh um at this table at a restaurant the four seasons hotel in seoul and eric schmidt's there and i asked i asked dennis like a lot of people are skeptical here what do you think is going to happen he says let me just tell you this system is continuing to learn and it's going to be better than people think it is and um and that just set things up for i mean just an incredible week it wasn't just about the fact that they had built this incredible system it was about the fact that the entire country was focused on this right in korea i remember having a pizza watch party at the berkeley ai lab streaming streaming them that first match it was just everybody was just like what's gonna happen um and we didn't even understand the game of ghosts we had no clue who was ahead we had to listen to the commentators get any sense of who might even be in the lead but we were just staring at it not understanding the game just so much tension so much excitement so now so yeah take that excitement in that one room okay and extrapolate that to an entire country okay that's what it was like right the whole country is focused on this because go with a national game in korea and you know it's you know it's one of their you know most important citizens who is competing against this machine and we all as people can relate to that right and this notion that you know machines are going to sort of overtake us and so as a human being you you pull for lee sidol and if you're korean you're pulling for him even more um and to watch you know this system um at first you know win the first game um uh which was shocking and then when the second one in this you know astounding way you could feel the air come out of an entire country right it was really i have to say it was really upsetting okay because um you know we're all humans right and we pull for for our fellow uh for our fellow person and then to watch um lee sito come back and win a game all right and um in a way that was just as astounding as the way the machine had won in game game two it was a remarkable thing um [Music] in in every respect ai had been very effective helping us out with machine translation speech recognition image recognition and so forth but but it doesn't even though those are very hard problems and in fact many would argue harder than than games they don't feel as hard to humans but go it feels hard and so you get this thing where hey the computer is doing something that we all think is hard and actually beating us at it which is so interesting it's true it's about it being hard but it's also about it's something that we all can relate to right we can understand the game right it's harder for people to wrap their head around a new advance in in machine translation um you know with a game it's very simple right you win or you lose um and and that's easy for for anyone to grasp and you could feel that you know in korea i mean 60 million people watched it in china right but also you could just feel you could feel things shift in the u.s right and among reporters um even among my editors at wired magazine they had been skeptical about all this you know seemingly crazy deep learning stuff that that my colleagues and i had been covering and we and we we were so intent on covering it because we could see where it was going my editors were skeptical and that was the moment where even they said oh okay we get it and and they get it because there's an important result there it showed what the technology can do but also it's easy to wrap your head around a cave and so it's interesting you reference china there because it comes up in in and just a couple chapters later what happens is some people argue that maybe still possibly in the moment not over the last 10-year span but maybe an in the moment even stronger go player is in china and i mean deepmind is is part of google at this point google has so much success with ai building products that are very popular in the us and the way i read chapter in the book and correct me if i'm wrong is that google had a strategy i hadn't heard about before reading about this where they thought hey internet was blocked in china we couldn't get that done but maybe ai is a new wave of things and maybe we can now start selling our products in china also and we're going to do that by doing a go event in china and from there you know we can grow our presence in china but it didn't play out the way they thought it would just tell more about that no it didn't and as fascinating as korea was you know i i i went to i went to china um as well as about a year later and and that was fascinating in its own right in a completely different way and the dynamic that you're talking about which is detailed in the book um is exactly is exactly right um in that that comes from conversations that were had at the very top of the company that they saw this as an opportunity right that the week in korea had been such uh you know a pr boon for google they thought we could recreate this in china you know we'll take this we'll take this machine to china and this will give us you know sort of the the currency we need um to to really make some headway in this country which we pulled out of you know just a few years before but uh it didn't work out the way they had planned and um it was a moment when um that you know the chinese government kind of woke up to what was happening and really thought to itself and this is all detailed in the book you know should we be supporting this company um that is is not a chinese company right um is it you know is a company based in the united states and should we should we be giving them the pr that they want here um and there's and i was there right there was a moment right before this happened where the chinese government said we are not going to broadcast this in the country um i talked to chinese reporters who were there they got um uh they got notices that said if you write about this you cannot use the word google right interesting that's so interesting that's what happened in the moment right it's like the curtain came down on this event and the only people in the country who knew what was really knew what was going on were the people who were there you know in this it's called a watertown this sort of ancient chinese town south of shanghai and you had to walk through metal detectors to get into this this uh this arena where this was held every morning and there were a relatively small number of people there and you got this feeling that you were the only person uh only only a few people in this in this enormous country who could really see what was happening it was it was a fascinating thing that's so amazing and it's i mean while while there is the you know from a google perspective no no success in china it's fair to say that many of the chinese companies have actually been really good at building ai technology themselves and you know the chinese public has access to ai technology just as much in the products that they're presented with just it's it's hosted by chinese companies not not by google it's at least the way it seems to be now right absolutely and one thing that's fascinating to me is that it wasn't like and a lot of people have gotten this wrong i think in the press it wasn't like you know china was unaware what was going on until that moment you know baidu is there when jeff hinton you know has that image net result right um you know it's so interesting to me that that china and baidu in particular have been involved in this from the beginning like we tend to think about um because this is the narrative promoted by these com american companies that they were there first but it's far more complicated than that and um you know this is this is an international revolution right it's not about um just american tech companies you know um we've talked about where the talent was it wasn't in the u.s it was elsewhere and china was involved from the very beginning and you know we've talked about a lot of fascinating people but there are so many fascinating people on that side of things um with um which we who we haven't talked about chile who was at microsoft and was born in china li dang uh who was at microsoft born in in china um you know the list goes on uh you know it's so interesting how this is this is by no means an american story well if anything it's be a canadian story in some sense right exactly [Music] we are dropping new interviews every week so subscribe to the robot brains on whichever platform you listen to your podcasts now at this point alphago is a way the public sees what's going on but the public actually subconsciously is experiencing ai everywhere like ai's by 2015 2016 ai is in our everyday lives i mean we we used to not be able to talk to our phones because they wouldn't recognize what we say rather type but now it can recognize what we say um it can recognize people in images help sort them it can do so many things and it's just embedded in in our everyday lives at least our digital lives at that at that moment in so many ways um but with ai becoming something real also a lot of problems came out right and your your next chapter kind of gets into that that actually they say ais were maybe not as good as people thought they were in terms of how accurate they are and and issues they have can see a bit more about that yeah i mean there are several threads to talk about here but one one thing that really comes to the fore is is the realization that because these systems train on human generated data right pictures we've taken pictures that we choose for our training set right we talk about imagenet and we're pooling these images someone has to choose that those images right someone has to choose the sounds that are used to train a system like siri that can recognize speech and what people start to realize is that these these systems we're building can be can be biased against women and can be biased against people of color if we're talking about systems that generate um uh you know media whether it's a sound or it's an image themselves we talked about you know these generative models that can create images um uh natural language systems that can generate text they can they can be toxic right they can say things um that we don't necessarily want them to say that can right they can be hurtful um and this starts to come to the fore and it's a really fascinating thing that that the industry is still struggling with we see this in headlines no just this you know you know just in the past few weeks where these giant companies are struggling how to deal with this and people are saying we need to think a lot harder about this because it's a it's inherent to the technology right if you're building a system that learns natural language from books that people have written from wikipedia articles from all sorts of constant the internet conversations it's inherently going to be biased because we as humans are biased right and there are holes in the way we see the world and we spew toxic language and if the systems are going to learn from us it's going to learn our our flaws as well as our strengths and in some sense what i like is that in some cases this common thread comes back again because this is something that people were not really questioning it was a blind spot i think it's fair to say um even though it's obvious in hindsight it was a blind spot and then these these three women women of color stepped in and saw that blind spot there was deb raji joy walamuwini and timot gebru who have been more in the news this year of course um because i mean the public seeing it now but they well start bringing to the four maybe they saw it soon but it started bringing to the force several years ago and your book describes how for example face recognition systems um were not performing the way one would expect can you say a little bit about that absolutely i mean i mean again just so many interesting people to talk about and so many interesting stories um deb raji's story is so interesting when she's at this this company called clarify and she starts to realize this right it's just you can just see that in our conversation we've had here today that so many people we're talking about are white men right and if you're a white men like myself white man like myself you're gonna have you're gonna have a certain view of the world right and and your view of the world is is going to factor in the in the way you build these systems what data you choose where your blind spots are and what dev realizes in the moment is that the data that they're using to train their face recognition systems and their content moderation system which is a system to kind of identify um toxic images is that they're biased right because the data that that the systems are trained on is biased and and she starts to realize this um and and others you mentioned uh tim neat and joy you know who start to build um or start to you know start to focus their research on this and look at these systems and look at the bias seeds and call attention to it it's a huge part of this story and it's the women who realized the blind spots and actually realized that yeah maybe it's not the terminators that are gonna we have to worry about today but look at how much damage this can already do today by just not being too naive about how we go about building ai systems right it's so true and i love that you bring up the terminator there because there's this great moment the book where tim knee you know she posts this manifesto almost to facebook right and she talks about that she says hold on you know i'm not worried about the terminator destroying us all i'm worried about this problem in the here and now right it's a great way to think about it um and it's fascinating to me um you know how we as a society still don't know how how to deal with this and these companies don't know how to deal with it it's such a hard problem right um uh and there's no good solution at this point and um and i'm i'm really interested to see how this plays out and how these companies are going to tackle this but um it's such an important issue i mean it's one way to look is no good solution and of course researchers are working on that but i mean the researchers have built a good amount of understanding of this and my sense is that the work by deb joined timmy has given a good amount of understanding of what's wrong with the current systems and we can already decide if we want to i mean some companies didn't i mean your book talks about amazon saying hey you're not evaluating our system the way it's supposed to be evaluated we have our own way of evaluating we're going to keep using it to amazon face recognition system and they went against that and i mean that's what you mean with i mean companies don't always understand it seems but it seems like we we start to have a while we have a long way to go start have a pretty good understanding at least of the vulnerabilities of these systems right and how we shouldn't just blindly deploy them the vulnerabilities are obvious right i mean it's just like there's no denying that right the problem is there it becomes a question you know of how you deal with that and what tim neat and and others are saying is we have to slow down and we have to think about what we're doing and we have to address the problem right um and you know that has to happen and um and you know i think these companies are still struggling with how to do that right people talk about what i meant was is that people talk about you know there is a solution right um it's not that simple right you know it's not easy to remove this problem that everybody sees right um again we talk about the over opt optimistic nature of these companies it's very easy to say we can solve this um well show me how to solve it right i don't think anyone has a good answer people talk about well we can use synthetic data or the like um you know uh we're still struggling with with how to deal with that and you know i was just talking to a guy you know jack clark uh who was it who did policy at open ai and um and you know he does this thing called the ai index where they sort of examine the state of the industry um and you know he points out that we just don't have the data we need to really analyze the bias in these systems and we need to work harder to to develop that right and to truly understand where the problems are and to solve them if we don't you know if we can't pinpoint this we can't solve it it seems like part of the problem of course is that with the data that's available money can be made by building ai systems a lot of money can be made but at the expense often of underrepresented groups um and i think that's where people like deb george timmy have started to jump in and say hey you're right you you pinpointed do you even realize what you're doing like it's like and now that you realize are you finally gonna are you finally gonna listen now that you know what's up or you're gonna keep keep the money train going right um and it's not just an eye problem it's also uh a policy and a decision like it's decisions you make right like with anything you make decisions what you decide as your priority in life or for your company right absolutely and again you pinpointed the issue right there's a profit motive here right you know these companies are designed to build technology and put it out right and there's tremendous amounts of money to be made so they have this incentive to put it out and then you know um you have people like tim neet and uh her collaborate google meg mitchell saying hold on you know we need to think about these other things and that's the clash that you're seeing right um and it's playing out in just big ways um and it's playing out in public even i mean it's one of the most important debates of the probably the most important debate in ai of the last year is is exactly centered around all that you got that exactly right so yeah i imagine if there's if there's a future version of the book there's going to be some chapters filling that part of the story how that's going to play out it's you're exactly right and a lot of work to be done hopefully it plays out the right way you're exactly right and you know um you know it's interesting in the book right there's this clash between tim neat and meg and amazon right it's exactly the same clash um it's just that it was with a different company right and and then you know fast forward a few months it's with their own company and what we haven't said here is that to meet and meg mitchell who ran the ethical um ai team at google they are no longer with the company right and you know that you know that has been painted as a situation at google and it is it's a situation at every company right this this this is something that microsoft is going to have to address facebook's going to have to address open ai who's in partnership with microsoft all these companies that we're talking about right this is the big issue yeah taught me about the mission of the company talk about mission of the company right you cover this thing called project maven which is i mean i think fascinating in itself can you say that what is project maven and what what happened there oh project may have been is is an effort to take these ideas and uh apply them inside the military okay um these are ideas that that uh can be extremely useful to say the least uh to the military right if you can recognize an object in an image that means you can you can help your your your drones fly on on their own that means you can identify targets on a battlefield it also means you can build autonomous weapons right not only identify the target um but fire at it right not to put to find a point on it right and what ended up happening is is not too long after the you know everything we've been talking about google started to work um uh on on a project inside the dod called project maven which confused a lot of people including a lot of people at the company right it was an effort and still is an effort to to build a system that can identify objects in drone footage right and some people have claimed over the years this is not for um for use with weapons um some people say it is um the the dod is certainly moving ahead with this and and their intent i've talked to people just in the past few weeks about this is to put this in the field right on the battlefield and you know at the very least there's a fine line between you know just using this to identify a target and firing at a target and so what you saw was the the the department of defense the military wanting this technology and they knew where it was right it's in companies like google we've talked about this at link that's that's where the progress is being made and google among others um amazon microsoft were involved in this as well as so many startups you know we're working on the dod with with this project well there was a big protest at google and a lot of employees took him and took issue with this right and um and you know it's a preview of of this this conflict we're seeing between google and tim neat and meg mitchell we talked about right where we've got people the companies saying we have a problem um and and there being this conflict between the leadership of the company and its employees um google ended up pulling out of that project uh not renewing their contract um and it's indicative of of something again that's bigger than just one company i mean the military wants to be ahead of course of other militaries and so they're going to try to find wherever the ai talent is to to help them out that's very complicated now ai has had its share of winters and what i thought were you know over promised under the liver and it goes quiet for a while what i found really intriguing in the book is that you hit this moment in 2018 i believe where people are starting to say well it's good for image recognition it's good for speech recognition machine translation it's all good but maybe it's over promised and under delivered because there's so many other things we want to do and just a few months after people start you know talking about that and making that like a topic like you know deep learning has its limits natural language processing has its major breakthroughs all right can you say a bit about that yeah i think this particularly fascinating we alluded this earlier that you know people call them universal language models right and essentially it's just a giant neural network that learns from text right it learns the ins and outs of of natural language like english by analyzing just reams of written english so self-published books thousands of books wikipedia articles other content from the internet these systems can analyze that find the patterns in it and then you know in some ways learn how language is pieced together and that can be used in a lot of ways right it can help the google search engine um deliver you search results the ones that you want understand your query and give you what you want but it can be used in in more powerful ways as well right this is the way we're starting to build chat bots right a system that can carry on a conversation and this is fascinating is that these these giant models they learn the in broad strokes the way that that language works in many ways and then you can apply that to these particular tasks like conversation right and essentially it can learn from the way that humans uh converse you know through um you know the conversations that are available you know across the internet we this is we converse on the internet through through chat services and like the system can learn to carry on a turn-by-turn conversation and it's not perfect yet right but we see we see real real progress in this area and and it's fascinating to see this play out as well and so maybe for people who aren't that familiar with ai in terms of where they might see it in their daily lives where it's under the hood as a reporter beyond just writing the book i mean what are some examples where people might be interacting with ai today well i mean the big ones i always bring up are your siri right when you speak in speak commands into your phone all right it can recognize what you say that that's a neural network at work there you see face recognition just on facebook right if you use facebook every day you know it can recognize bases and that's how how that works but the google search engine as i alluded to right those giant language models are already helping helping that system work and work in more pointed ways anyone who's used google over the past several years has seen the way it improves and that's in part because of these ideas we're talking about um but like the list goes on um you know there's been so much talk and we can go into this more if you like about self-driving cars right right this technology is fundamental to to self-driving cars as well and if we didn't have this neural network idea that we've been discussing um the the progress of self-driving cars will be very very different to say the least right a self-driving car or a self-piloting drone which we also alluded to they use those neural networks that do image recognition fundamentally right how do you recognize a pedestrian and how do you recognize a street side it's a neural network all these things are drawing on this one idea that we've been talking about and that's so fascinating right this is one idea neural networks which market is the one idea that maybe the brain also has but of course we don't really understand the brain fully this neural network that powers all these applications if we look at what we see today i mean if you ask people what is ai going to what do you want from ai a lot of people would imagine ai would bring a home robot that you know does laundry maybe cooks cleans up after dinner yeah we're not we're not seeing a home robot i mean we have a roomba that can scoot around the floor but we don't have a butler type home robot right so it's this is in some sense there seems to be this big gap between how ai for digital has really come of age already but then ai for physical ai robotics is what we like to call it ai robotics um is it seems harder the physical world seems harder because self-driving car has been promised for many years and progress is made but and so i'm curious how how how you see that kind of transition into physical yeah i think it's fundamental to understanding what's going on here and the self-driving car thing is a great example and and it goes back to this this notion of hype we talked about right um there was such hype around self-driving cars and what i heard as a reporter and what so many reporters repeated you know ad infinitum was that self-driving cars were going to be here by 2020 right i remember a lot of 2020 mentioned a lot of 20 20 like we're in we're in 2021 i don't know about you but if i walk out beside my door in berkeley i'm not going to see a self-driving car and um you know what in what happened um that i think people understand is that if you're a reporter you got a call that said hey come take a ride in my self-driving car and you and you and you got in and you circled mountain view right you circled a few blocks near google's campus and you got out and you thought my goodness right this is this is real we're gonna have these things you know i don't know in a few hours well the reality is that um that little loop around mountain view california uh is is contained and controlled and there is so much chaos in our daily lives that you and i and you know instinctually know how to deal with after years of living that machines cannot deal with right it's about that extreme uncertainty and we talk about results like like go or chess as complex as go is it's a contained universe right and our world is not contained in the same way and there's there's just so many things that can that can happen to me when i step out of my front door and get in a car that i know how to deal with uh that a machine uh cannot right and there's a hope that as we continue to develop these things and and in part because of these systems that learn from data like if we get enough data the machine can deal with all that uncertainty but we're not there yet right it's a really hard thing to do and i think you know there has been enormous pro you know this of course is what you work in enormous product progress in robotics but we need to think about where that you know where that's really working versus where it's still aspiration right and there are certain things where it can work really well and there are other places where it can yeah i distinctly remember sitting in in my office at berkeley with you probably fall 2019 and um you had just covered this fascinating result from open ai the the hand solving the rubik's cube and it's just mind-boggling because well it's hard for humans but also for a robot hand the five-fingered hand controlling that reliably and all the contact forces that was a very hard problem and they cracked it and so very exciting moment for robotics to see that happen but at the same time we're talking about how that's still very different from self-driving cars where things are far less predictable it's you're in this new environment and then i remember you saying at the time if you can if you can show me something robotics in the real world uh ai robotics or ai powered robotics in the real world i'll be very curious to see it um and i remember thinking okay well that's that's what we're working on at cover and of course we're trying to bring these ideas into real world in in warehouses and i i remember thinking well okay that's going to be interesting because we might be able to do that and and then indeed i mean new york times came and checked it out and it seems like we're starting to get there with with ai robotics at least in more structured places like warehouses uh coming of age even if driving is still a little too unstructured yeah the warehouse thing is is is a good way to think about it right and that's not something the average person necessarily thinks about but but again especially during this time all right uh the time of this pandemic i think people can relate to the fact that they rely so heavily on companies to ship them stuff right mm-hmm i mean it's just like every day right it seems like a new package shows up on my door seemingly you know every other day and you have these giant distribution centers that have that have to do that for you right you have all sorts of goods coming in and then they've got to go out to the right places and the way it works today is you you know for the most part is you have people who do that right they have to sort through all the stuff that's coming in um get it to the right spot sometimes it waits in the warehouse for a while sometimes it goes out and you know you got to retrieve it and you know there's so many moving parts there and you need people to do that right it's it may not be obvious to people but if you put a like a random collection of stuff on a table you and i can easily sort through it right pick stuff up put them in a place that's a very hard task for a machine it has been traditionally um and there's tremendous demand again right a company like amazon or fedex or ups and other retailers we're at this moment where they need more people to do this than are really available in a lot of areas and like i've seen that you know like here in the valley where there are all these distribution centers you know kind of outside san francisco where there's not a lot of people or at least there's not they have trouble hiring enough people so there's this real demand for machines that can do that task right and you know this right because it's the problem you're working on um but what we're seeing um is is the ability to take these ideas a neural network apply them to that you know robotic iron that can sift through these these kinds of piles of stuff they call it the picking problem right if you can do that um that you know that's something that say an amazon again or a fedex really wants um and it's a marriage of where the technology is progressing but also demand right sometimes there's a mismatch between the technology and what people want it to do in this case it lines up i mean we all want to keep getting things at our doors and it also leads to um one of one of my personal favorite quotes in in the book of course where you describe um how jeff hinton describes when he saw the covering ai system in this case doing the robotic pick and place and he describes as the alpha go moment for robotics where robots are going beyond repeated motion and actually interacting with the situation at hand it's really true and it you know it's funny how you know you get again you get into the ai community and you realize there's there's such a wide range of opinion and there's so much infighting and some people believe in reinforcement learning as they call it which is like basically it's learning by trial and error right you know you you you can do this of course with robotics and the picking problem you you just have the system whether it's in the digital world or actually in the physical world learning just by trying to pick stuff up and failing and trying again and and and that sort of extreme um reinforcement learning idea right this done at a scale that systems have not been capable of in the past that's something that hinton didn't really believe in right um he would he and he does this in his inamidable you know incredibly funny way where he'll sort of dismiss you know deep mind with a sweep of his hand and the deep learning idea but even he has come to recognize the value of this idea because he knows that we now have a at least part of the processing power that we need to do that right it's something we didn't have in the past and you've seen so many interesting results um around that kind of trial and error learning that we didn't have in the past because we've got such enormous amounts of uh of processing power it's interesting to see his change in view there and then to hear him talk about it because it's just it's just flat out funny absolutely and i mean one way to think of it is that that data is noisier because it's trial and error data so it's it's not as high signal as as the typical data we would often train with but if you get enough noisy data ultimately you can still extract the signal right and right now towards the end of the book um we kind of have this i would say celebration of these early pioneers with something called the touring what is the touring award and yeah what what is that all about the turn award the best way to think about it is it's the nobel prize of computing right it's given it's given yearly to um the computer scientists or someone in a related field um and um in in recent years i mean for people aren't familiar with it you know it is a huge deal in the tech industry right um absolutely um you can't understate that um and what we saw um in 2019 was jeff hinton and john lecoon and joshua benjio was another collaborator of theirs another person who really believed in this idea of a neural network their contributions were recognized they won the turing award um as a as a threesome um and uh you know i was lucky enough to be there that the the night that they received it and um you know and also talked to them um when it was announced and i did a piece in the in the times and um it was it was fascinating to watch hinton's reaction um in that moment when he received the award we've talked a lot about hinton and his sense of humor and you know there's so many other things we haven't talked about um uh his personal quirks and and also his physical problems right he's got his back problem which is mind-boggling in its own right and it plays into this story and it's amazing how his sort of he does he literally does not sit down that's the first sentence in the book the man does not sit down um because this back problem he has he realized that he had the only way to control his back problem um was was to stop saying now that means he can't drive it means he can't fly because commercial airlines make you sit during during takeoff and landing and this plays into his effort you know to evangelize uh you know this technologies building and bring it to these giant companies he's got to travel you know across continents across oceans and how is he going to do this right that's its own story but then you know you get to this moment where he receives his award and it's a really emotional moment right and there's this other thread involving his wife um you know who was very ill uh with cancer as he's going through all this and he talks about this in a moment it's a really amazing amazing moment and an emotional moment and that's the other thing you know we've talked about so many big ideas we've talked about the people we've talked about the humor of these people but there's a lot of other emotions wrapped up in all this right absolutely you know and relationships and and not only conflict but but heartbreak and sadness and um it you know it's it's as as remarkable as story as any other yeah so that's where the book ends but i'm curious having talked with so many ai researchers pioneers in the last couple years to write this book and and and also to write for the new york times about the latest breakthroughs how do you extrapolate to the future what do you if you were asked to make some what do you think is going to happen next well i mean one thing i i do see is is where the progress is continuing right this is what you look for at least in my mind as a tech reporter is where you see where you see things paying off right and [Music] since i finished the book we've already seen a really important result one thing we haven't talked about is is alpha fold right which is you know this this result out of deep mind again um in the biological sciences this is essentially about drug discovery it's about applying these ideas um uh to proteins which are essential to the way the human body operates and and the way that we fashion medicines and drugs um that can deal with things that go wrong in the body right viruses um and other conditions and and again it mirrors that alphago result they had a result there that people thought was decades away a result on a problem that that scientists had worked worked on for decades and had failed and that people thought was still decades away and and they essentially cracked it and that's an area where i'm i'm i'm really looking um i'm looking at things hard and i think you're going to continue to see progress what would be the impact what's the impact of doing really well at this kind of protein folding prediction how is it going to affect our lives i mean again um it's it's something that we've been dealing with as a society for the past year right this pandemic right this pandemic arrives um it's a new situation a new condition and you have to decide how to deal with it right what existing medicines can we use um with people who have covet right how can we help help them in the moment when they're trying to deal with with the medicines that have already been approved for use and then on top of that how do we build a vaccine um that that that can help us in a larger way it's about building new ones um that can work in those situations it's something we've dealt with as a society over the past year and the hope there is that the next time this happens we can deal with it quicker right um that we will have the tools not only to repurpose the medicines we have but to build new medicines like a vaccine so that's massive right if that can happen that that that might be the biggest thing ai will have done until till that moment what are some other things that that you see well that you know we've talked about the these these uh giant language models that's an area where i'm really interested to see where things go you've got so many companies working on on that they've got this enormous problem the bias problem that we talked about that they have to deal with um but that's you know that's where a lot of progress is um where is that going to go it's in another area area and that's bigger than people think right i just wrote a piece about companies they're building you know autonomous flying drones for the military right as we pointed out the military really wants this stuff there are now companies that are working hand in hand with the military to do it where is that going to go and and and what are we going to do about all the problems there and the concerns there um and this is not a a u.s thing it's a global thing um uh how are we going to deal with that as a society those are the three big areas right biological sciences natural language and robotics of course robotics has many challenges ahead as as we know uh as you alluded to with self-driving and and might start a more structured environment so i mean people might not want to hold their breath maybe for self-driving for for next year just yet it's a good point right with all this stuff right is that as much as we see progress there are these huge questions um that we have to answer right as a society right it's and um and we're struggling with all this and that's why i think that the book is important and this field that we've been talking about is so important is that these these are huge questions we're grappling with and it's not one question it's many questions in all these areas right um the you know natural language and and the bias issues and the the toxic language issue there right and and um and when you get to robotics and self-driving cars it you know it's about safety right how do you how can you ensure that these systems um are safer than human drivers and and and of course um you know we have to figure out how to make these things you know not only you know drive around the block um but ensure they don't kill people right right you know it's it's as fundamental as that you know i originally you know put together my pitch for the book and and sold it to my publisher when i was required magazine and you know the the pitch was great and and and in my mind and and it was essentially what the first half of the book became but um the book got so much more interesting as time went on and we realized that all these questions were facing us right and that's what the second half of the book is about is these big questions um but you know the other thing i always go back to is that it's about these these people right as i pointed out my father was an electrical engineer and i always wanted to write a book about about people like him right he worked on something different he was a program ibm but one of the things he worked on that people will know is is the universal product code right the barcode that's on on on every grocery that gets scanned when you reach the cashier you know that was developed at ibm and my father um uh worked on that project he was he helped test that project and um he has you know these amazing stories about that i grew up on these stories and that's what i'm trying to do with this book is is focus on on the people building uh the technology and not only the great stories they have and um and the great moments they live through um but the the questions are grappling with themselves personally you know how do they see the way the world is is embracing or raising concerns over the technologies that they personally build right uh that's what interests me so kade this has been just an absolutely wonderful conversation thank you so much for joining us i'd like to remind everyone that if you want to learn more about ai and people behind it cademet's book is called genius makers the mavericks who brought ai to google facebook and the world and you can buy it from any of your favorite book sellers online really enjoyed it peter as always great talking to you sam here thank you so much kade [Music] do [Music] you

Channel: The Robot Brains Podcast

Views: 7,549

Keywords: Cade Metz, Pieter Abbeel, artificial intelligence, facebook, google, deepmind, geoff hinton

Id: vvy_jcArbhY

Length: 121min 5sec (7265 seconds)

Published: Tue Sep 21 2021