George Hotz: Comma.ai, OpenPilot, and Autonomous Vehicles | Lex Fridman Podcast #31

the following is a conversation with George Hotz he's the founder of comma AI a machine learning based vehicle automation company he is most certainly an outspoken personality in the field of AI and technology in general he first gained recognition for being the first person to carry or unlock an iPhone and since then he's done quite a few interesting things at the intersection of hardware and software this is the artificial intelligence podcast if you enjoy it subscribe on YouTube give it five stars on iTunes supported on patreon or simply connect with me on Twitter at lex friedman spelled fri d-m a.m. and i'd like to give a special thank you to Jennifer from Canada for her support of the podcast on patreon merci beaucoup Jennifer she's been a friend and an engineering colleague for many years since I was in grad school your support means a lot and inspires me to keep this series going and now here's my conversation with George Hotz do you think we're living in a simulation yes but it may be unfalsifiable what do you mean by unfalsifiable so if the simulation is designed in such a way that they did like a formal proof to show that no information can get in and out and if their hardware is designed to for the anything in the simulation to always keep the hardware in spec it may be impossible to prove whether we're in a simulation or not so they've designed it such there's the closed system you can't get outside of the system well maybe it's one of three worlds we're either in a simulation which can be exploited we're in a simulation which not only can't be exploited but like the same things too about VMs I'm a really well-designed VM you can't even detect if you're in a VM or not that's brilliant so where it's yeah so the simulation is running in a virtual machine but now in reality all VMs have wasted the fact that's the point I mean is it yeah you've done quite a bit of hacking yourself and so you should know that really any complicated system will have ways in and out so this isn't necessarily true going forward I spent my time away from comma I learned and said dependently typed like it's a language for writing math proofs and if you write code that compiles in a language like that it is correct by definition the types check it's correct and so it's possible that the simulation is written in a language like this in which case yeah yeah but that can't be sufficiently expressive a language like that all weekend it can be yeah okay well so all right so the simulation doesn't have to be tiring complete if it has a scheduled end date looks like it does actually with entropy and you know I don't think that a simulation that results in something as complicated in universe would have a formal proof of correctness right as as possible of course we have no idea how good their tooling is and we have no idea how complicated the universe computer really is it may be quite simple it's just very large right it's very it's definitely very large but the fundamental rules might be super simple yeah Conway's gonna live kind of stuff right so if you could hack it so imagine the simulation that is hackable if you could hack it what would you change about the you know like how would you approach hacking a simulation the reason I gave that talk I by the way I'm not familiar with the talk he gave I just read that you talked about escaping the simulation yeah like that so maybe you can tell me a little bit about the theme and the message there - it wasn't a very practical talk about how to actually escape a simulation it was more about a way of restructuring an us-versus-them narrative if we continue on the path we're going with technology I think we're in big trouble like as a species and not just as a species but even as me as an individual member of the species so if we could change rhetoric to be more like to think upwards like to think about that we're in a simulation and how we could get out already we'd be on the right path what you actually do once you do that while I assume I would have acquired way more intelligence in the process of doing that so I'll just ask that so the the thinking upwards what kind of ideas what kind of breakthrough ideas do you think thinking in that way could inspire and what did you say upwards upwards into space are you thinking sort of exploration in all forms the space narrative that held for the modernist generation doesn't hold as well for the postmodern generation what's the space narrator we're talking about the same space the dimensional space like going a little ace is like building like yuan mosque like we're gonna build rockets we're gonna go to Mars we're gonna colonize the universe and the narrative your friend was born in the Soviet Union you're referring to the race to space the race to space explore okay that was a great modernist narrative it doesn't seem to hold the same weight in today's culture I'm hoping for good postmodern narratives that replace it so think let's think so you work a lot with AI so the eyes one formulation of that narrative there could be also I don't know how much you do in VR and they are yeah that's another eye I know less about it but every time I play with it and our research is fascinating that virtual world are you are you interested in the virtual world I would like to move to a virtual reality in terms of your work no I would like to physically move there the apartment I can rent in the cloud is way better in the apartment I can rent in the real world well it's all relative isn't it because others will have very nice departments too so you'll be inferior in the virtual world that's not how I view the world right I don't view the world I mean it's very like like almost zero-sum issue a to view the world say like my great apartment isn't great because my neighbor has one - no my great apartment is great because like look at this dishwasher man yeah you just touch the dish and it's washed right and that is great in and of itself if I have the only apartment or if everybody had the apartment I don't care so you have fundamental gratitude the the world first learned of Geo ha George Hotz in August 2007 maybe before then but certainly in August 2007 when you were the first person to unlock carry unlock an iPhone how did you get into hacking what was the first system you discovered vulnerabilities for and broke into so that was really kind of the first thing I had I had a book in in 2006 called grey hat hacking and I guess I realized that if you acquired these sort of powers you could control the world but I didn't really know that much about computers back then I started with electronics the first iPhone hack was physical card work um you had to open it up and pull an address line high and it was because I didn't really know about software exploitation I learned that all in the next few years and I got very good at it but back then I knew about like how men chips are connected to processors and he knew about software and programming he didn't didn't know I'll really see you the view of the world and computers was physical was the most hard work actually if you read the code that I released with that in August 2007 it's atrocious the language was it a C say yes and in a broken sort of state machine SC I didn't know how to program man so how did you learn to program what was your journey cuz I mean we'll talk about it you've live streams from your programming man this is a chaotic beautiful mess how did you arrive at that years and years of practice I interned at Google after the summer after the iPhone unlock and I did a contract for them where I built hardware for for Street View and I wrote a software library to interact with it and it was terrible code and for the first time I got feedback from people who I respected saying you know like don't write code like this now of course just getting that feedback is not enough the way that I really got good was I wanted to write this thing like that could emulate and then visualize like armed binaries because I wanted to hack the iPhone better and I didn't like that I couldn't like see what that I couldn't single step through the processor because I had no debugger on there especially for the low level things like the boot ROM in the bootloader so I tried to build this tool to do it and I built the tool once and it was terrible I built the tool second times it was terrible I built the tool third time this by the time I was at Facebook it was kind of okay and then I built the tool fourth time when I was a Google intern again in 2014 and that was the first time I was like this is finally usable how do you pronounce this kira-kira yeah so it's essentially the most efficient way to visualize the change of state of the computer as the program is running that's what I mean by debugger yeah it's a timeless debugger so you can rewind just as easily as going forward think about if you're using gdb you have to put a watch on a variable if you want to see if that variable changes and Kure you can just click on that variable and then it shows every single time when that variable was changed or accessed think about it like get for your computers uh the run lock so there's like a deep log of of the state of the computer as the program runs and you can rewind why isn't that maybe it is maybe you can educate me what isn't that kind of debugging used more often ah because the tooling is bad well two things one if you're trying to debug chrome chrome is a 200 megabyte binary that runs slowly on desktops so that's going to be really hard to use for that but it's really good to use for like CTFs and for boot roms and for small parts of code so it's it's hard if you're trying to debug like massive systems what's the CTF and what's the boot ROM the boot ROM is the first code that executes it's the minute you give power to your iPhone okay and CTF were these competitions that I played capture the flag to capture the flag I was going to ask you about that what are those LaVette I watched a couple videos on YouTube those look fascinating what have you learned about maybe at the high level of vulnerability of systems from these competitions the like I feel like like in the heyday of CTFs you had all of the best security people in the world challenging each other and coming up with new toy exploitable things over here and then everybody okay who can break it and when you break it you get like there's like a file on the server called flag and then there's a program running listening on a socket that's vulnerable so you write an exploit you she'll and then you cat flag and then you type the flag into like a web-based scoreboard and you get points so the goal is essentially to find an exploit in the system that allows you to run shell to run arbitrary code on that system that's one of the categories that's like the PO noble category vulnerable yeah horrible it's like you know you pwned the program you are it's a program yeah yeah you know for personally I apologize I'm gonna I'm gonna say it's because I'm Russian but maybe you can help educate me some video game like misspell to own way back in the Mia and there's just I wonder if there's a definition I'll have to go to urban dictionary for it okay so what was the heyday seat yeah by the way but was it what decade are we talking about I think like I mean maybe I'm biased because it's the era that that that I played but like 2011 to 2015 because the modern CTF scene is similar to the modern competitive programming scene you have people who like do drills you have people who practice and then once you've done that you've turned it lesson to a game of generic computer skill and more into a game of okay you memorize you you drill on these five categories and then before that it wasn't it didn't have like as much attention as it had I don't know they were like I won $30,000 ones in Korea for one of these competitions oh crap they were they were that so that means I mean money's money but that means there was probably good people there exactly yeah are the challenges human constructive or are they grounded in some real flaws and real systems usually they're human constructed but they're usually inspired by real flaws what kind of systems are imagined is really focused on mobile like what has vulnerabilities these days is it does primarily mobile systems like Android everything does No yeah of course the price has kind of gone up because less and less people can find them and what's happened in security is now if you want to like jailbreak an iPhone you don't need one exploit anymore you need nine nine chained together what women yeah Wow okay so it's really so what's the but what's the benefit speaking higher level philosophically about hacking I mean it sounds from everything I've seen about you you just love the challenge and you don't want to do anything you don't want to bring that exploit out into the world and doing the actual let it run wild you just want to solve it and then you go on to the next thing oh yeah I mean doing criminal stuffs not really worth it and I'll actually use the same argument for why I don't do defense for why I don't do crime if you want to defend a system say the system has ten holes right if you find nine of those holes as a defender you still lose because the attacker gets in through the last one if you're an attacker you only have to find one out of the ten but if you're a criminal if you log on with a VPN nine out of the ten times but one time you forget you're done because you're caught okay because you only have to mess up once to be caught as a criminal yeah that's why I'm not a criminal but okay let me uh that's having a discussion with somebody just at a high level about nuclear weapons actually why we're having blowing ourselves up yet and my feeling is all the smart people in the world look at the distribution of smart people smart people are generally good and then this other person I was talking to Sean Carroll the physicist and you were saying no good and bad people are evenly distributed amongst everybody my sense was good hackers are in general good people and they don't want to mess with the world what's your sense I'm not even sure about that like I have a nice life crime wouldn't get me anything but if you're good and you have these skills you probably have a nice life too right like you can use the father things but is there an ethical is there some is there a little voice in your head that says well yeah if you could hack something to where you could hurt people and you could earn a lot of money doing it though not hurt physically perhaps but disrupt her life in some kind of way it is there a little voice that says um what two things one I don't really care about money so like the money wouldn't be an incentive the thrill might be an incentive but when I was 19 I read crime and punishment right that was another that was another great one that talked me out of ever really doing crime Oh cuz it's like that's gonna be me I'd get away with it whatever just went in my head even if I got away with it you know and then you do crime for long enough you'll never get away with it that's right in the end that's a good reason to be good I wouldn't say good I just say I'm not bad you're a talented programmer and a hacker in a good positive sense of the word award you've played around found vulnerabilities in various systems what have you learned broadly about the design of systems and so on from that from that whole process you learn to not take things for what people say they are but you look at things for what they actually are yeah I understand that's what you tell me it is but what does it do man and you have nice visualization tools to really know what it's really doing oh I wish I'm a better programmer now than I was in 2014 I said Kira that was the first tool that I wrote that was usable I wouldn't say the code was great I still wouldn't say my code is great so how was your evolution as a programmer except practice he went he started with C at which point did you pick up Python because you're pretty big and Python though now yeah in uh in college I went to Carnegie Mellon when I was 22 um I went back I'm like I'm gonna take all your hardest CS courses we'll see how I do right like did I miss anything by not having a real undergraduate education took operating systems compilers AI and they're like a freshman reader math course and operating says some of these some of those classes you mentioned actually they're great at least one the 2012 circuit 2012 operating systems and compilers we're two of the best classes I've ever taken my life because you write an operating system and you write a compiler I wrote my operating system in C and I wrote my compiler in Haskell but classical well somehow I picked up Python that semester as well I started using it for the CTS actually that's when I really started to get into CTF and CTF you're all to race against the clock so I can't write things and say oh there's a clock component so you really want to use the programming language you can be fastest than 48 hours pone as many of these challenges you can pone yeah you got like a hundred points a challenge whatever team gets the most you were both the Facebook and Google for a brief stint yeah well the project zero actually at Google for five months where you develop kara what was project zero about in general speak what what just curious about the security efforts in these companies well product zero started the same time I I went there what what years are there 2015 2015 so that was right at the beginning of project it's small it's Google's offensive security team I'll try to give I'll try to give the best public facing explanation that I can so the idea is basically these vulnerabilities exist in the world nation states have them some high powered bad actors have them sometime people will find these vulnerabilities and submit them in bug bounties to the companies but a lot of the companies don't really care it only fix the bug there's no it doesn't hurt for there to be a vulnerability so project zero is like we're gonna do it different we're going to announce a vulnerability and we're going to give them 90 days to fix it and then whether they fix it or not we're gonna drop the drop the zero day oh wow we're gonna drop the weapon that's so cool that is so cool I love that deadlines though that's so cool give him real deadlines yeah and I think it's done a lot for moving the industry forward I watched your coding sessions on the stream downline you code things up basic projects usually from scratch I would say sort of as a programmer myself just watching you that you type really fast and your brain works in both brilliant and chaotic ways I don't know if that's always true but certainly for the live streams so it's it's interesting to me because I'm more I'm much slower and systematic and careful and you just move I mean probably an order of magnitude faster some curious is there a method to your madness is this just who you are there's pros and cons there's pros and cons to my programming style and I'm aware of them like if you ask me to like like get something up and working quickly with like an API that's kind of undocumented I will do this super fast because I will throw things at it until it works if you ask me to take a vector and rotate it 90 degrees and then flip it over the XY plane I'll spam program for two hours and won't get it all because it's something that you could do with a sheet of paper think through design and then just you really just throw stuff at the wall and you get so good at it that it usually works I should become better at the other kind as well sometimes I'll do things pathetically it's nowhere near as entertaining on the twitch streams I do exaggerate it a bit on the edge games as well the twitch streams I mean what do you want to see a game or you want to see actions permit me right I'll show you a PM for programming yes I recommend people go to I think I watched I was probably several hours you put like I've actually left you programming in the background while I was programming because you made me you it was it was like watching a really good gamer it's like energizes you because you're like moving so fast it so it's it's awesome it's inspiring and so it made me jealous that like because my own program is inadequate in terms of speed Oh as I was like so I'm twice as frantic on the live streams as I am when I code without oh it's super entertaining so I I wasn't even paying attention to where you were coding which is great it's just watching you switch windows and VAM I guess is driven screen I've developed a workflow Facebook and talk about how do you learn new programming tools ideas techniques these days what's your like methodology for learning new things so I wrote for comma the distributed file systems out in the world are extremely complex like if you want to install something like like like Saif Saif is I think the like open infrastructure to should be a file system or there's like newer ones like seaweed FS but these are all like 10,000 plus line projects I think some of them are even 100,000 line and just configuring them as a nightmare so I wrote I wrote one um it's 200 lines and it's it uses like nginx to the live servers and has low master server that I wrote and go and the way I go this if I would say that I'm proud per line of any code I wrote maybe there's some exploits that I think are beautiful and then this this is 200 lines and just the way that I thought about it I think was very good and the reason it's very good is because that was the fourth version of it that I wrote and I had three versions that I threw away you mentioned you see go I ready go yeah and go so is that a functional language I forget what goes they go is Google's language right I'm a functional it's some it's like in a way it's C++ but easier it's it's strongly typed it has a nice ecosystem erotic when I first looked at it I was like this is like Python but it takes twice as long to do anything yeah now that I've open pilot is migrating to sea but it still has large Python components I now understand why Python doesn't work for large code bases and why you want something like Oh interesting so why why doesn't Python work for so even most speaking for myself at least like we do a lot of stuff basically demo level work with autonomous vehicles and most of the work is Python yeah why doesn't Python work for large code bases because well lack of type checking is a big errors creeping yeah and like you don't know the compiler can tell you like nothing right so everything is either you know like like syntax errors fine but if you misspell a variable and Python the compiler won't catch that there's like linters that can catch it some other time there's no types this is really the biggest downside and then will Python slow but that's not related to it well maybe the kind of related to its that's lacking so what's what's in your toolbox these days is a Python what else go I need to move on something else but my adventure interdependently type languages I love these languages they just have like syntax from the 80s what do you think about JavaScript yes thanks Nick tomorrow typescript javascript is the whole ecosystem is unbelievably confusing NPM updates a package from zero to two to zero to five and that breaks your babble linter which translates your es5 into es6 which doesn't run on so why do I have to compile my JavaScript again huh it may be the future though if you think about I mean I've embraced JavaScript recently because just like I've continually embraced PHP it seems that these worst possible languages live on for long is that cockroaches never die yeah well it's in the browser and it's fast it's fast yeah it's in the browser and compute mites they become you know the browser it's unclear what the role the browser's in terms of distributed computation in the future so javascript is definitely here to stay yeah interesting if Tom's vehicles will run on JavaScript one day I mean you have to consider these possibilities well all our debug tools are JavaScript we actually just open-source them we have a tool Explorer which you can annotate your dis engagements and we have tool cabana which lets you analyze the canned traffic from the car so basically any time you're visualizing something about the log you using javascript yeah well the web is the best UI toolkit by far yeah um so and then you know what you're voting in JavaScript we have a react guy he's good he acts nice let's get into it so let's talk to Thomas vehicles you found it comma a let's at a high level how did you get into the world the vehicle automation can you also just for people who don't know tell the story of comma yeah sure so I was working at this AI startup and a friend approached me and he's like dude I don't know where this is going but the coolest applied AI problem today is self-driving cars I'm like well absolutely do you want to meet with UI mosque and he's looking for somebody to build a vision system for auto pilot this is when they were still on ap one they were still using mobile I kneel on back then was looking for a replacement and he brought me in and we talked about a contract where I would deliver something that meets mobile eye level performance I would get paid twelve million dollars if I could deliver it tomorrow and I would lose 1 million dollars for every month I didn't deliver yeah so I was like ok this is a great deal this is a super exciting challenge you know what even if it takes me 10 months I get two million dollars it's good maybe I can finish up in five maybe I don't finish it at all and I get paid nothing and I'll work for twelve months for free so maybe I just take a pause on that I'm also curious about this because I've been working on robotics for a long time and I'm curious to see a person like you just step in and sort of somewhat naive but brilliant right so that's though that's the best place to be because you basically full-steam take on a problem how confident how from that time because you know a lot more now at that time how hard do you think it is to solve all of autonomous driving I remember I suggested to Elon in the meeting I'm putting GPU behind each camera to keep the compute local this is an incredibly stupid idea I leave the meeting 10 minutes later and I'm like I could have spent a little bit of time thinking about this problem was I would just send all your cameras to one big GPU you're much better off doing that oh sorry you said behind every camera you have a small GPU I was like oh I'll put the first few layers of my comm there Oh like why did I say that that's possible it's possible but it's a bad idea it's not obviously a bad idea pretty obvious but whether it's actually a bad idea or not I left that meeting with Elon like beating myself up I'm like why did I say something stupid yeah you haven't given I'm at least like thought through every aspect yes he's very sharp too like usually in life I get away with saying stupid things and then kind of course alright right away he called me out about it and like usually in life I get away with saying stupid things and then like people will you know people a lot of times people don't even notice and I'll like correct it and bring the conversation back but with Elon it was like nope like okay well that's not at all why the contract fell through I was much more prepared the second time I met him yeah but in general huh how hard did you think it is like 12 months is uh-oh is it tough timeline oh I just thought I'd clone mob like you three I didn't think I'd solve level five self-driving or anything so the goal there was to do lane-keeping good good link keeping I saw my friend showed me the outputs from a mobile I in the office from a mobile I was just basically two lanes at a position of a lead car mm-hm like I can I can gather a dataset and train this net in in weeks and I did well first time I tried the implementation of mobile I and the test I was really surprised how good it is it's quite incredibly good because I thought it's just because I've done a lot of computation I thought it'd be a lot harder to create a system that that's stable so I was personally surprised you know have to admit it because I was kind of skeptical before trying it because I thought it would go in and out a lot more it would get disengaged a lot more and it's pretty robust so what how how hard is the problem we need to when you tackled it I think a p1 was great like Elon talked about dis engagements on the 405 down in LA we'd like the lane marks were kind of faded and the mobile eye system would drop out uh like I had something up and working that I would say was like the same quality in three months same quality but how do you know you you say stuff like that yeah confidently but you can't and I love it but well the question is you can't you're kind of going by feel because he not solely absolutely like like I would take I hadn't I borrowed my friends Tesla yeah I would take ap one out for a drive yeah and then I would take my system out for a dry and seems reasonably like the same so the four or five how hard is it to create something that could actually be a product that's deployed I mean I've read an article or you on this respondent said something by you saying that to build autopilot is is more complicated than a single George Hotz a level job how hard is that job to create something that would work across the globe Lee what are the global ease the challenge but Elon followed that up by saying it's gonna take two years in a company of ten people yeah and Here I am four years later with a company of twelve people and I think we still have another two to go two years so yeah so what do you think what do you think about the hottest is progressing with autopilot v2 v3 I think we've kept pace with them pretty well I think navigator autopilot is terrible we had some demo features internally of the same stuff and we would test it and I'm like I'm not shipping this even as like open-source software to people what do you think is do Consumer Reports does a great job of describing it like when it makes a lane change it does it worse than a human you shouldn't ship things like autopilot open pilot they Lane keep better than a human if you turn it on for a stretch of highway like an hour long it's never gonna touch a lane line human will touch probably a lane line twice you just inspired me I don't know if you're grounded and data on that I read labor okay but no but that's interesting uh I wonder actually how often we touch Lane lines in general like a little bit cuz it is okay I could answer that question pretty easily with the common data side yeah I'm curious I've never answered it I don't know yeah I just - is like my person it feels right that's interesting because every time you touch the lane that's the source of a little bit of stress and kind of lane-keeping is removing that stress that's all to me the big the biggest value-add honestly is just removing the stress of having to stay in lane and I think honestly I don't think people fully realize first of all that that's a big value add but also that that's all it is and that not only I find it a huge value add I drove down when we moved to San Diego I drove down our Enterprise rent-a-car and I missed it so I missed having the system so much it's so much more tiring to drive without it it's it is that Lane centering that's the key feature yeah and in a way it's the only feature that actually adds value to people's lives and autonomous vehicles today way mode does not add value to people's lives it's a more expensive lower slower uber maybe someday it'll be this big cliff where it adds value but I don't usually do this vessei I haven't talked to is that this is good because I haven't I have intuitively but I think we're making it explicit now I I actually believe that really good lane-keeping is a reason to buy a car will be a reason to buy a car is a huge value add I've never until we just started talking about it haven't really quite realized that that I've felt with elan chase of level four is not the correct chase it was on because you should just say Tesla has the best as if from a testing perspective say Tesla has the best lane-keeping coming I should say coming I is the best link keeping and that is it yeah yeah does do you think well you have to do the longitudinal as well you can't just Lane keep you have to do a cc but a cc is much more forgiving than lanky especially on the highway oh by the way are you uh calming eyes camera only correct oh no we use the radar we from the car you were able to get to open it um we can't do a camera only now it's gotten to the point but we leave the radar there is like a it's it's fusion now okay so let's maybe talk through some of the system specs on the hardware or what it what's what's the hardware side of what you're providing what's the capabilities in the software side would open pilot and so on so open pilot as the the box that we sell that it runs on it's a phone in a plastic case it's nothing special we sell it without the software so you're like you know you buy the phone it's just easy it'll be easy setup but it's sold with no software open pilot right now is about to be 0.6 when it gets to 1.0 I think we'll be ready for a consumer product we're not gonna add any new features we're just gonna make the lane-keeping really really good so what do we have right now it's a snapdragon 820 say so many IMX 298 forward-facing camera driver monitoring camera and she's a selfie cam on the phone and a can transceiver biffle's little thing calls pandas and they talk over USB to the phone and then they have three canvases that they talk to the car one of those campuses is the radar CANbus one of them is the main car CANbus and the other one is the proxy camera CANbus we leave the existing camera in place so we don't turn a DB off right now we still turn a TV off if you're using our longitudinal but we're gonna fix that before 1.0 you got it wow that's cool so in its can both way so how are you able to control vehicles so we proxy the vehicles that we work with already have Lane Keeping Assist system so Lane Keeping Assist can mean a huge variety of things it can mean it will apply a small torque to the wheel after you've already crossed a lane line by a foot which is the system in the older Toyotas versus like I think Tesla still calls it Lane Keeping Assist where it'll keep you perfectly in the center of the lane on the highway you can control like you would in joystick the cars these so these cars already have the capability of drive-by-wire so is it is it trivial to convert a car that it operates with it open pile is able to control the steering Oh a new car or a car that we so we have support now for 45 different makes of cars what are one of the cars general mostly Hondas and Toyotas we support almost every Honda and Toyota made this year and then a bunch of GM's bunch of Subarus which it doesn't have to be like a Prius it could be Coral as well okay the 2020 Corolla is the best car with open pilot it just came out there the actuator has less lag than the older Corolla I think I started watching video with your eye the way you make videos is awesome literally the dealerships streaming stream for an hour yeah and basically like if stuff goes a little wrong you're just like you just go with it yeah I love it what's real yeah that's real that's that's it's that's so beautiful and it's so in contrast to the way other companies would put together a video like that how do I like to do it like good I mean if you become super rich one day is successful I hope you keep it that way because I think that's actually what people love that kind of genuine oh it's all that has value to me yeah my money has no if I sell out to like make money and I sold out it doesn't matter what do I get yacht I don't I got and I think Tesla's actually has a small inkling of that as well with autonomy day they did reveal more than I mean of course there's marketing communications you can tell but it's more than most companies will reveal which is I hope they go towards a direction more other companies GM Ford oh Jessa Tesla's gonna win level 5 they really are so let's talk about it you think you're focused on level 2 currently currently we're gonna be one to two years behind Tesla getting to level five okay we're interested right we're into it you're in I'm just saying once Tesla gets it we're one to two years behind I'm not making any timeline on when Tesla's that's right you did that's brilliant I'm sorry Tesla investors if you think you're gonna have an autonomous robot taxi fleet by the end of the year yes that's all bet against that so that what do you think about this the most level four companies are kind of just doing their usual safety driver during full autonomy kind of testing and then Tesla does basically trying to go from lane-keeping to full autonomy what do you think about that approach how successful would it be a ton better approach because Tesla is gathering data on a scale that none of them are they're putting real users behind the behind the wheel of the car it's I think the only strategy that works the incremental well so there's a few components to test approach that's that's more than just incrementally you spoke with is the one is the software so over-the-air software updates necessity I mean way more ease have those - those aren't but there was differentiating from the automaker's right no link keeping assist systems have no cars with lane keeping system have that except Tesla yeah and the other one is the data the other direction which is the ability to query the data I don't think they're actually collecting as much days people think but the ability to turn on collection and turn it off so I'm both in the robotics world in the the psychology human factors world many people believe that level to autonomy is problematic because of the human factor like the more the task is automated the more there's a vigilance decrement you start to fall asleep you start to become complacent start texting more and so on do you worry about that because if we're talking about transition from lane-keeping to full autonomy if you're spending eighty percent of the time not supervising machine do you worry about what that means to the safety of the drivers one we don't consider open pilot to be 1.0 until we have 100% driver monitoring you you can cheat right now our driver monitoring system there's a few ways to cheat it there pretty obvious we're working on making that better before we ship a consumer product that can drive cars I want to make sure that I have driver monitoring that you can't cheat what's like a successful driver monitoring system look like it's keep its is it all buzz just keeping your eyes on the road um well a few things so that's what we went with it first for driver monitoring I'm checking I'm actually looking at where your head is looking but cameras know about my resolution eyes are a little bit hard to get well head is this big I mean that is good and actually a lot of it just as psychology wise to have that monitor constantly there it reminds you that you have to be paying attention but we want to go further we just hired someone full-time to come onto the driver monitoring I want to detect phone in frame and I want to make sure you're not sleeping how much does the camera see of the body this one not enough not enough the next one everything what's interesting fish Atkins we have we're doing just data collection that real-time but fish eye is a beautiful mouth being able to capture the body and the smartphone is really like the biggest problem I'll show you I can show you one of the pictures from from our finder system awesome so you're basically saying the driver monitoring will be the answer to that um I think the other point that the original paper is is good as well you're not asking a human to supervise a machine without giving them meat they can take over at a time right our safety model you can take over we disengage on both the gas or the brake we don't disengage on steering I don't feel you have to but we disengage on gas or brake so it's very easy for you to take over and it's very easy for you to re-engage that switching should be super cheap yeah the cars that require even autopilot requires a double press that's almost I said I like that yeah and then then the cancel um to cancel in autopilot you either have to press cancel which no one knows where that is so they press the brake but a lot of things you don't you want to press the brake you want present ass so you should cancel on gas or wiggle the steering wheel which is bad as well wow that's brilliant I haven't heard anyone articulate at that point I like what this is all I think about it's because I think I think actually Tesla has done a better job than most automakers at making that frictionless but you just described that it could be even better I love super cruise as an experience once it's engaged yeah I don't know if you've used it but getting the thing to try to engage him yeah I've used this of Germany's super cruise a lot so what's their thoughts on the super Cruiser system in June disengage super cruise and it falls back to ACC so my car's like still accelerating it feels weird otherwise when you actually have super cruise engaged on the highway it is phenomenal we bought that Cadillac we just sold it but we bought it just to like experience this and I wanted everyone in the office to be like this is what we're striving to build GM pioneering with the driver monitoring you know you like their driver monitoring system it has some bugs if there's a sun shining back year it'll be blind to you by overall mostly yeah that's so cool you know the stuff that's uh I don't often talk to people that because it's such a rare car unfortunately they bought one yes possibly for us we lost like by 25k the deprecation but a Philips worth it I was very pleasantly surprised that GM system was so innovative and really that wasn't advertised much wasn't talked about much yeah and I was nervous that it would die that they would disappear my eyes did they put it on the wrong car they should've put it on the bolt and not some weird Cadillac that nobody bought I think that's gonna be into they're saying at least is going to be into their entire fleet so what do you think about it if as long as we're on the driver monitoring what do you think about you know I must claim that driver monitoring is not needed normally I love his claims that one is stupid that one is stupid and you know he's not gonna have his level five fleet by the end of the year hopefully he's like okay I was wrong I'm gonna add driver monitoring because when these systems get to the point that they're only messing up once every thousand miles you absolutely need driver monitor so let me play Delta because I agree with you but let me play devil's advocate so one possibility is that without driver monitoring people are able to monitor the self-regulate monitor themselves you know that so your idea is seeing all the people sleeping in decimals uh yeah well I'm a little skeptical of all the people sleeping in Tesla's because I have I've stopped paying attention to that kind of stuff because I want to see real data there's too much glorified it doesn't feel scientific to me so I want to know you know what how many people are really sleeping in Tesla's vs. sleeping I've I was driving here sleep-deprived in a car with no automation I was falling asleep I agree that it's high P it's just like you know what if you under I've am wondering I think I rented a my last autopilot experience was I rented a model three in march and drove it around the wheel thing is annoying and the reason the wheel thing is annoying we use the wheel thing as well but we don't disengage on wheel for Tesla you have to touch the wheel just enough you should trigger the torque sensor to tell it that you're there but not enough as to disengage it which don't use it for two things you disengage one wheel you don't have to that whole experience Wow beautiful put that all those elements even if you don't have driver monitoring that whole experience needs to be better driver monitoring I think would make I mean I think super cruise is a better experience once it's engaged over autopilot I think super cruise is our transition to engagement and disengagement are significantly worse yeah so there's a tricky thing because if I were to criticize super cruise is uh it's a little too crude and uh I think it's like six seconds or something if you look off-road you'll start warning you it's some ridiculously long period of time and just the way it I think it's basically it's a binary chili adapter it yeah it's it just needs to learn more about you and used to communicate what it sees about you more like I'm not you know Tesla shows what it sees about the external world it would be nice the supercruise would tell us what it sees about the internal world it's even worse than that you press the button to engage and it just says super cruise unavailable yeah why why yeah that transparency is good we've renamed the driver monitoring packet to driver state service state we have car state packet which has the state of the car driver state packet which I stay the driver so what does itah make their BAC must be do you think that's possible with computer vision absolutely so to me it's an open question I don't haven't looked into too much they actually had quite seriously looked at the literature it's not obvious to me that from the eyes and so on you can tell you might need to stuff from the car as well yeah you might need how they're controlling the car right and that's fundamentally at the end of the day what you care about you but I think especially when people are really drunk they're not controlling the car nearly smoothly as they would look at them walking right there the car is like an extension of the body so I think you could totally detect and if you could fix people who drunk distracted asleep if you fix those three yeah this is that's huge so what are the current limitations of open pilot what are the main problems that still need to be solved um we're hopefully fixing a few of them in 0-6 we're not as good as auto pilot at stop cars so if you're coming up to a red light at like 55 so it's the radar stopped car problem which is responsible to auto pilot accidents it's hard to differentiate a stopped car from a like signpost yes that ecology um so you have to fuse you have to do this visually there's no way from the radar data to tell the difference maybe you could make a map but I really believe in mapping at all anymore um really what you don't believe in mapping no so you basically the open pilot solution is saying react to the environment is just like human doing beings and then eventually when you want to do navigate on open pilot I'll train the net to look at ways all runways in the background I'll train a car using GPS at all we use it to crown trees we use it to very carefully ground treat the paths we have a stack which can recover a relative to 10 centimeters over one minute and then we use that to ground truth exactly where the car went in that local part of the environment but it's all local how are you testing in general just for yourself like experiments stuff all right were you were you located San Diego San Diego yeah okay Oh what you basically drive around there then collect some data and watch on Florence we have a simulator now and we have our simulators really cool our simulator is not it's not like a unity based simulator our simulator lets us load in real estate what I mean we can load in a drive and simulate what the system would have done on the historical data ooh nice interesting so what yeah right now we're only using it for testing but as soon as we start using it for training what's your feeling about the real world versus simulation do you like simulation for training if this moves to training Chuck we have to distinguish two types of simulators right there's a simulator that light is completely fake I could get my car to drive around in GTA mm-hmm um I feel that this kind of simulator is useless you're never there's so many my analogy here is like okay fine you're not solving the computer vision problem but you're solving the computer graphics problem right and you don't think you can get very far about creating ultra realistic graphics no because you can create ultra realistic graphics of the road now create alter a realistic behavioral models of the other cars oh well I'll just use my self-driving no you won't you need real you need actual human behavior because that's what you're trying to learn the dead driving does not have a spec the definition of driving is what humans do when they drive whatever way mode does I don't think it's driving right well I think if you win more than others its if there's any useful reinforcement learning I've seen it used quite well I study pedestrians a lot too is try to train models from real data of how pedestrians move and try to use reinforcement learning models to make pedestrians move in human-like ways by that point you've already gone so many layers you detected a pedestrian did you did you hand code the feature vector of their state did you guys learn anything from computer vision before deep learning well okay you know I feel like this is a perception to you is the sticking point does that mean what what's what's the hardest part of the stack here there is no human understandable feature vector separating perception and planning that's the best way I can I can put that there is no so it's all together and it's it's a that's a joint problem so you can take localization localization and planning there is a human understandable feature vector between these two things I mean okay so I have like three degrees position three degrees orientation and those derivatives maybe those second derivatives right that's human understandable that's physical the between perception and planning um so like way Moe has a perception stack and then a planner um and one of the things way matters right is they have a simulator that can separate those two they can like replay their perception data and test their system which is what I'm talking about about like the two different kinds of simulators there's the kind that can work on real data and is the kind of can't work on real data now the problem is that I don't think you can hand code a feature vector right like like you have some lists of like well here's my list of cars on the scenes here's my list of pedestrians in the scene this isn't what humans are doing what are humans doing global some something you're saying that's too difficult to handle I'm saying that there is no state vector given a perfect I could give you the best team of engineers in the world to build a perception system and the best team to build a planner all you have to do is define the state vector that separates those two I'm missing the state vector that separates those two what do you mean so what is the output of your perception system I'll put it the perception system it's theirs okay well there's several ways to do it one is this lamp components localization the other is drivable area drivable space drivable space and then there's the different objects in the scene and different objects in the scene over time maybe to give you input to then try to start modeling the trajectories of those objects sure that's it I can give you a concrete example of something you missed what's that so say there's a bush in the scene humans understand that when they see this bush that there may or may not be a car behind that bush drivable area and a list of objects does not include that humans are doing this constantly at the simplest intersections so now you have to talk about occluded area right right but even that what do you mean by occluded okay so I can't see it well if it's the other side of a house I don't care what's the likelihood that there's a car in that occluded area right and if you say okay we'll add that I can come up with 10 more examples that you can't add certainly occluded area would be something that simulator would have because it's simulating the entire you know occlusion is part of it a part of a vision stack pleasures that what I'm saying is if you have a hand engineered if your perception system output can be written in a spec document it is incomplete yeah idem you know certainly it's it's hard to argue with that because in the end that's going to be true yes I'll tell you what the output of our perception system is was that it's a thousand it's a thousand twenty four dimensional vector training underling oh no not it's a thousand twenty four dimensions of who knows what because its operating on real data yeah yeah and that's the perception that's the perception stake right think about a think about an autoencoder four phases alright if you have an autoencoder four phases and you say it has 256 dimensions in middle and I'm taking a face over here and projecting it to a face over here yeah can you hand label all 256 of those dimensions well no but those are generated automatically but they but even if you tried to do it by hand could you come up with a spec for your and between your encoder and your decoder no no because that's not it is it wasn't designed but there no no but if you could design it if you could design a face Reconstructor system could you come up with a spec no but I think we're missing here a little bit I think the the you're just being very poetic about expressing a fundamental problem of simulators that they're going to be missing so much that the feature vector would just look fundamentally different from in the simulated world in the real world I'm not making a claim about simulators I'm making a claim about the spec division between perception and planning and planning even in your system just in general right just in general if you're trying to build a car that drives if you're trying to hand code the output of your perception system like saying like here's a list of all the cars in the scene here's a list of all the people here's a list of the included areas here's a vector of drivable areas insufficient and if you start to believe that you realize that what Wayman crews are doing is impossible currently what we're doing is the perception problem it's converting the scene into a chessboard you yeah and then you reason some basic reasoning around that chessboard yeah and you're saying that really there's a lot missing there first of all why are we talking about this cuz isn't this a full autonomy is this something you think about oh I want to win self-driving cars so you're really your definition of win includes level of fool five level five I don't think level four is a real thing I want to build I want to build the alphago of driving so so alphago is really end to end yeah is uh yeah it's end to end and do you think this whole problem is those that also kind of what you're getting at with the perception and the planning is that this whole problem the right way to do it is really to learn the entire thing I'll argue that not only is it the right way it's the only way that's going to exceed human performance well certainly true for go everyone who tried to hand code go things built human inferior things and then someone came along and wrote some 10,000 line thing that doesn't know anything about go that beat everybody it's 10,000 lines true in that sense the the open question then that maybe I can ask you is uh driving is much harder than go the open question is how much harder so how because I think the AH mosque approach here with planning and perception it's similar to what you're describing which is really turning into not some kind of modular thing but really do formulate is a learning problem and it solves a learning problem of scale so how many years put one is how many years would it take to solve this problem or just how hard is this freaking problem well the cool thing is I think there's a lot of value that we can deliver along the way I think that you can build lane-keeping assist actually plus adaptive cruise control plus okay looking at ways extends to like all of driving yeah most of driving varies oh your adaptive cruise control treats red lights like cars okay so let's jump around with you you mentioned that you didn't like navigate an autopilot yeah what advice how would you make it better do you think as a feature that if it's done really well it's a good feature I think that it's too reliant on like hand coded hacks for like how does navigate an autopilot do a lane change it actually does the same lane change every time and it feels mechanical humans do different lane changes human sometime will do a slow one sometimes do a fast one navigate an autopilot at least every time I used it it did the identical language how do you learn I mean this is a fundamental thing actually yeah is uh the braking and an accelerating something that's still test the probably does it better than most cars but it still doesn't do a great job of creating a comfortable natural experience and navigate on autopilot just lane changes an extension of that so how do you learn to do natural lane change so we have it and I can talk about how it works so I feel that we have the solution for lateral but we don't yet have the solution for longitudinal there's a few reasons longitudinal is harder than lateral the lane change component the way that we train on it very simply is like our model has an input for whether it's doing a lane change or not and then when we train the end-to-end model we hand label all the lane changes because you have to I struggled a long time about not wanting to do that but I think you have to because you order the training data for the train data right well we actually we have an automatic ground truth or which automatically labels all the lane changes was that possible to automatically label interest yeah and detect the lane I see when it crosses it right I don't have to get that that high percent accuracy but it's like 95 good enough now I set the bit when it's doing the lane change in the end-to-end learning and then I set it to zero when it's not doing a lane change so now if I wanted to do a lane change a test time I just put the bit to a 1 and I'll do later yeah but so if you look at the space of lane change you know some percentage not a hundred percent that we make as humans is not a pleasant experience because we messed some part of it up yeah it's nerve-racking to change even look at the seizure des accelerate how do we label the ones that are natural and feel good you know that's the because that's your ultimate criticism the current Oh navigate not apologies doesn't feel good well the current navigator on autopilot is a hand coded policy written by an engineer in a room who probably went out and tested it a few times on the 280 probably a more a better version of that but yes that's how we would have written it a comment yeah Tesla they tested it and it might have been two engineers yeah no but so if you learn the lane change if you learn how to do a lane change from data just like just like you have a label that says lane change and then you put it in when you want to do the lane change it'll automatically do the lane change that's appropriate for the situation now to get it the problem of some humans do bad lane changes we haven't worked too much on this problem yet it's not that much of a problem in practice my theory is that all good drivers are good in the same way and all bad drivers are bad in different ways and we've we've seen some data to back this up well beautifully put so you just basically if that's true yeah hypothesis then you know task is to discover the good drivers um the good drivers stand out because they're in one cluster and the bad drivers are scattered all over the place and your net learns the cluster yeah that's uh so you just learned from the good drivers and they're easy to cluster we learned from all of them and that automatically learns the policy that's like the majority but we'll eventually probably afterthought so if that theory is true I hope it's true because the the counter theory is there is many clusters maybe but rarely many clusters of good drivers because if there's one cluster of good drivers you can at least discover a set of policies you can learn a set of policies which would be good universally yeah that would be a nice that would be nice if it's true and you're saying that there are some evidence that let's say lane changes can be clustered into four clusters right right there's this finite level of I would argue that all four of those are good clusters all the things that are random are noise and probably bad and which one of the four you pick or maybe it's Tanner maybe it's twenty you can learn them it's context dependent it depends on the scene and the hope is it's not too dependent on the driver yeah the hope is that it all washes out the hope is that there's that the distribution is not bimodal the hope is that it's a nice gas man so what advice would you give to Tessa how to fix how to improve navigate an autopilot the lessons you've learned from Kamiya the only real advice I would give to Tesla is please put driver monitoring in your cars with respect to improvement you can't do that anymore I said to interrupt but you know there's a practical nature of many of hundreds of thousands of cars being produced that don't have a good driver facing camera the model 3 has a selfie cam is it not good enough did they not have put IR LEDs for night that's a good question but I do know that the is fisheye in its relatively low resolution so it's really not this I he wasn't it wasn't designed for Arman you can hope that you can kind of scrape up and and and have something from it yeah but put it in today put it in today today every time I've heard Carpathia talk about the problem and talking about life software 2.0 and how the machine learning is gobbling up everything I think this is absolutely the right strategy I think that he didn't write navigate on autopilot I think somebody else did and kind of hacked it on top of that stuff I think what Carpathia says wait a second why did we hand code this lane change policy with all these magic numbers we're gonna learn it from data they'll fix it they already know what to do there well that that's that's Andres job is to turn everything into a learning problem and collect a huge amount of data the the reality is though not every problem could be turned into a learning problem in the short term in the end everything would be a learning problem the reality is like if you want to build alpha vehicles today it will likely involve no learning and that's that's the the reality is so at which point does learning start it's the crutch statement that lidar is a crutch on which point will learning get up to part of human performance it's all over human performance and imagenet classification under ivan is the question still it is a question I'll say this I'm I'm here to play for 10 years I'm not here to try to I'm here to play for 10 years and make money along the way I'm not here to try to promise people that I'm gonna have my l5 taxi Network up and working in two years do you think those mistake yes what do you think there was the motivation behind saying that other companies are also promising alpha vehicles with their different approaches in 2020 2021 2022 if anybody would like to bet me that those things do not pan out I will I will bet you even money even money I'll bet you as much as you want so are you worried about what's going to happen because you're not in full agreement on that I was going to happen when 2022 21 come around and nobody has fleets of autonomous vehicles no you can look at the history if you go back five years ago they were all promised by 2018 and 2017 but they weren't that strong of promises I mean Ford really declared pretty that I think not many have declared as as like definitively as they have now these dates well okay so let's separate l4 and l5 do I think that it's possible for way mo to continue to kind of like like hack on their system until it gets to level 4 in Chandler Arizona yes knows no safety driver Chandler Arizona yeah but by OSI which year are we talking about oh I even think that's possible by like 2020 2021 but level 4 Chandler Arizona not level 5 New York City level 4 meaning some very defined streets it works out really well very defined streets and then these streets are pretty empty if most of the streets are covered in way MOS we mo can kind of change the definition of what driving is hmm right if your self-driving network is the majority of cars in an area they only need to be safe with respect to each other and all the humans will need to learn to adapt to them now go drive in downtown New York oh yeah that's already you can talk about autonomy in like like fun farms it already works great because you can really just follow the GPS line so what does success look like for comm AI what what are the milestones like where you can sit back with some champagne and say we did it boys and girls well it's never over yeah but don't be let's drink champagne everything straight so what is a good what are some wins um a big milestone that we're hoping for by mid next year is profitability of the company and we're gonna have to revisit the idea of selling a consumer product but it's not gonna be like the comma one when we do it it's gonna be perfect open pilot has gotten so much better in the last two years we're gonna have a few a few features we're gonna have a hundred percent driver monitoring we're gonna disable no safety features in the car um actually I think it'd be really cool we're doing right now our project this week is we're analyzing the data set and looking for all the AEP triggers from the manufacturer systems we have a better data set on that than the manufacturers how much does how many does Toyota have ten million miles of real-world driving to know how many times they're AUB triggered so let me give you cuz yes right financial advice yeah cuz I work with a lot of automakers and one possible source of money for you which I'll be excited to see you take on is basically selling the data so which is something that most people are not selling in a way we're here here at automaker but creating we've done this actually at MIT not for money purposes but you could do it for significant money purposes and make the world a better place by creating a consortium where automakers would pay in and then they get to have free access to the data and I I think a lot of people are really hungry for that and would pay significant amount of money for it here's the problem with that I like this idea all in theory he'd be very easy for me to give them access to my servers and we already have all open source tools to access this data it's in a great format we have a great pipeline but they're gonna put me in the room with some business development guy mm-hmm and I'm gonna have to talk to this guy and he's not gonna know most of the words I'm saying I'm not willing to tolerate that okay but I think I agree with you I'm the same way but you just tell them the terms and there's no discussion needed if if I could just tell them the terms yeah and then like all right who wants access to my data I will sell it to you for let's say you want to go on a subscription I'll sell you 400 a month any 100k mo 100k month I'll give you access to the data subscription yeah yeah I think that's kind of fair came up with that number off the top of my head if somebody sends me like a three line email where it's like we would like to pay a hundred K month to get access to your data we would agree to like reasonable privacy terms of the people who are in the data set I would be happy to do it but that's not gonna be the email the email is gonna be hey do you have some time in the next month where we can sit down and we can I don't have time for that we're moving too fast yeah you could politely respond to that email but not saying I don't have any time for your yeah you say oh well unfortunately these are the terms and so this is we try to we brought the cost down for you in order to minimize the friction of education after here's the whatever it is 1 2 million years dollars a year and you have access and it's not like I get that email from like but okay am I gonna reach out am I gonna hire a business development person who's gonna reach out to the automaker's no way yeah okay if they reached into me I'm not gonna ignore the email I'll come back with something straight yeah if you're willing just pay honeycomb all the facts they don't man I'm happy to to set that up that's what my engineering time but actually quite insightful view you're right yeah probably because many of the automakers are quite a bit of old-school yeah there will be need to reach out and they want it but they they'll need to be some some communication you right mobile eye circuit 2015 had the lowest R&D spend of any chip maker like purpur and you look at all the people who work for them and it's all business development people because the car companies are impossible to work with yeah so you're you have no patience for that and you're you're legit Android huh I have something to do right like like it's not like it's not like I don't like I don't mean to like be a dick and say like I don't have patience for that but it's like that stuff doesn't help us with our goal of winning self-driving cars if I want money in the short term if I showed off like the actual like the learning tech that we have it's it's somewhat sad like it's years and years ahead of everybody else's not so maybe not Tesla's I think Tesla has similar stuff to us actually yeah I think Tesla's similar stuff but when you compare it to like what the Toyota Research Institute has you're not even close to what we have no comment but I also can't I have to take your comments I ain't into ative Lee believe you but I have to take it with a grain of salt because I mean you you are an inspiration because you basically don't care about a lot of things that other companies care about you don't try to in a sense like make up stuff so to drive a valuation you're really very real and you're trying to solve the problem and admire that a lot what I don't necessarily fully can't trust you on I do respect it's like how good it is right I can only but I also know how bad others are and so I'll say I'll say two things about don't trust but verify right I'll say two things about that one is try get in a twenty twenty Corolla and try open pal 0.6 when it comes out next month I think already you'll look at this and you'll be like them this is already really good and then I could be doing that all with hand labelers and all with with like like the same approach that like Mobileye uses when we release a model that no law has the lanes in it that only outputs a path mm-hmm then think about how we did that machine learning and then right away when you see and that's gonna be an open pilot that's gonna be an open pilot before 1.0 when you see that model you'll know that everything I'm saying is true because how else did I get that model good one of the things too about the simulator oh yeah yeah this is super exciting that's super exciting and uh but like you know I listened to your talk with Kyle and Kyle was originally building the the after market system and he gave up on it because of technical challenges yeah because of the fact that he's gonna have to support twenty to fifty cars we support forty five because what is he gonna do when the manufacturer ABS system triggers we have alerts and warnings to deal with all of that in all the cars and how is he going to formally verify it well I got ten million miles of data it's probably better it's probably better verified than the spec yeah I'm glad you're here talking to me this is I'll remember this day is this interesting if you look at Kyle's from from Cruz I'm sure they have a large number of business development folks and you work with he's working with GM you could work with agro a I working with Ford it's interesting because chances that you fail business-wise like bankrupt are pretty high yeah and and yet it's the Android model is you're actually taking on the problem so that's really inspiring I mean well I have a long-term way for kamma to make money too and one of the nice things when you really take on the problem which is my hope for autopilot for example is things you don't expect ways to make money or create value that you don't expect will pop up oh I've known how to do it since kind of 2017 is the first time I said it well which part to know it to know how to do which part our long-term plan is to be a car insurance company insurance yeah I love it yeah yeah what I make driving twice is safe not only that I have the best date is that you know who statistically is the safest drivers and oh oh we see you we see you driving unsafely we're not going to insure you and that that causes a like bifurcation in the market because the only people who can't get common insurance or the bad drivers Geico can insure them their premiums crazy higher premiums are crazy low would win contracts take over that whole market okay so if we win if we went but that's I'm saying like how do you turn comma into a ten billion dollar company is that that's right so you you know a musk who else who else is thinking like this and working like this in your view who are the competitors are there people seriously I don't think anyone that I'm aware of as seriously taking on lane-keeping you know like to worse a huge business that turns eventually into full autonomy that then creates yeah like that creates other businesses on top of it and so on thinks insurance thinks all kinds of ideas like that do you know who anyone else thinking like this not really that's interesting I mean it my sense is everybody turns to that in like four or five years like Ford once the autonomy doesn't feel fall through but at this time Elon to the iOS by the way he paved the way for all I was not i OS true I would not be doing comma AI today if it was not for those conversations with Elon and if it were not for him saying like yeah I think he said like well obviously we're not gonna use Leiter we use cameras humans use cameras so what do you think about that how important is lidar everybody else is on l5 is using lidar what are your thoughts on his provocative statement that lidar is a crutch see sometimes we'll say dumb things like the driver monitoring thing but sometimes we'll say absolutely completely 100% obviously true things yeah of course lidar is a crutch it's not even a good crutch you're not even using it they're using it for localization yeah which isn't good in the first place if you have to localize your car to centimetres in order to drive like yeah they're not drive it currently not doing much machine learning I thought polite our data meaning like to help you in the tasks of general tasks of perception the main goal of those light hours on those cars I think is actually localization more than perception or at least that's what they use them for yeah that's true if you want to localize two centimeters you can't use GPS the fanciest GPS in the world can't do it especially if you're under tree cover and stuff flatter I can do it pretty easily see really they're not taking on I mean in some research they're doing they're using it for perception but and they're certainly not which sad they're not fusing it well lay vision they do use it for perception I'm not saying they don't use it for perception but the thing that they have vision based and radar based perception systems as well you could remove the lidar and and and keep around a lot of the dynamic object perception you want to get centimeter accurate localization good luck doing that with anything else so what should Cruz lame-o do like what would you be your advice to them now anyway Mo's actually there's I mean they're doing they're serious way mo out of all of them equate so serious about the long game if everybody fell five is a lot is requires fifty years I think when will be the only one left standing at the end with the forgiving the financial backing if they have Google box um I'll say nice things about both lame-o and Cruz let's do it nice is good way mo is by far the furthest along with technology way mo has a three to five year lead on all the competitors um if that if the way mo looking stack works mm-hmm maybe three year lead if the way mo looking stack works they have a three year lead now I argue that way mo has spent too much money to recapitalize to gain back their losses in those three years also self-driving cars have no network effect like that yeah goober has a network effect you have a market you have drivers and you have riders self-driving cars you have capital and you have riders there's no network effect if I want to blanket a new city in self-driving cars i buy the off-the-shelf Chinese knockoff self-driving cars and I buy enough up from the city I can't do that with drivers and that's why Ober has a first mover advantage that no self-driving car company will can you uh disentangle that a little bit uber you're not talking about uber the autonomous vehicle number you talked about the uber cars okay yeah I'm over I open for business in Austin Texas listen I need to attract both sides of the market I need to both get drivers my platform and riders on my platinum and I need to keep them both sufficiently happy right riders aren't going to use it if it takes more than five minutes for an uber to show up drivers aren't gonna use it if they have to sit around all day and there's no riders so you have to carefully balance a market and whenever you have to carefully balance a market there's a great first mover advantage because there's a switching cost for everybody right the drivers and the riders would have to switch at the same time let's even say that you know um let's say Luber shows up in Luber somehow you know agrees to do things that add a bigger you know you know we're just gonna we've done it more efficiently right Luber is only takes five percent of a cot instead of the ten percent that Hooper takes no one is gonna switch because the switching cost is higher than that five percent so you actually can in markets like that you have a first mover advantage yeah autonomous vehicles of the level five variety have no first mover advantage if the technology becomes commoditized say I want to go to a new city look at the scooters it's gonna look a lot more like scooters every person with a checkbook can blanket a city in scooters and that's why you have 10 different scooter companies yeah which one's gonna win it's a race to the bottom it's terrible market to begin because there's no market for scooters and scooters don't get a say and whether they want to be bought and deployed to a city or not right so the yeah we're gonna entice the scooters with subsidies and deals so whenever you have to invest that capital that's it doesn't it doesn't come back yeah that they can't be your main criticism over the way mo approach oh I'm saying even if it does technically work even if it does technically work that's a problem yeah I don't know I if I were to say I I would I would say you're already there I haven't even thought about that but I would say the bigger challenge is the technical approach so way most cruises and not just the technical approach but of creating value I still don't understand how you beat uber the the human driven cars in terms of financially it doesn't it doesn't make sense to me that people want to want to get an autonomous vehicle I don't understand how you make money in the long term like real long-term but it just feels like there's too much capital investment needed oh and they're gonna be worse than ubers because they're gonna they're gonna stop for every little you know thing everywhere um actually a nice thing about Cruz that was my nice thing about wait another three years that it wasn't nice oh that's three years technically ahead of everybody their tech stack is is great my nice thing about Cruz is GM buying them was a great move for GM for 1 billion dollars GM bought an insurance policy against way mo they put Cruz is three years behind way mo hmm that means Google will get a monopoly on the technology for at most three years and technology works you might not even be right about the three years it might be less might be less crews actually might not be that far behind I don't know how much way mo has waffled around or how much of it actually is just that long tail yeah okay if that's the best you could say there's some nice things it that's more of a nice thing for GM that that's a smart insurance policy it's just more insurance policy I mean I think that's how I I can't see crews working out any other for crews to leapfrog way mo would really surprise me yeah so let's talk about like the underlying assumptions of everything is we're not going to leapfrog Tesla Tesla would have to seriously mess up for us because you're okay so the way you leapfrog right is you come up with an idea or you take a direction perhaps secretly that the other people aren't taking and so cruise way mo even Aurora no Aurora tzuke's is the same stack as well they're all the same codebase even and they're all the same DARPA urban challenge codebase so the question is do you think there's a room for brilliance and innovation there that will change everything like say okay so I'll give you examples it could be if revolution and mapping for example that allow you to map things do HD maps of the whole world all weather conditions really well or revolutionist simulation to where the the what you said before becomes incorrect that kind of thing I knew room for breakthrough innovation um what I said before about oh they actually get the whole thing well I'll say this about we divide driving into three problems and I actually haven't solved the third yet but I have an idea how to do it so there's the static the static driving problem is assuming you are the only car on the road right right and this problem can be solved 100% with mapping and localization this is why farms work the way they do if all you have to deal with is the static problem and you can statically schedule your machines right it's the same as like statically scheduling processes you can statically schedule your tractors to never hit each other on their paths all right because then you know the speed they go at so so that's the static driving problem Maps only helps you with the static driving problem yeah the question about static driving yeah you just made it sound like it's really easy that was really easy how easy how well because the whole drifting out of lane when when Tesla drifts out of lane is failing on the fundamental static driving problem Tesla is drifting out of lane the static driving problem is not easy for the world the static driving problem is easy for one route and one route in one weather condition with one state of lane markings and like no deterioration no cracks in the road I'm assuming you have a perfect localizer so that's all for the weather condition and me the lane marking condition that's the problem is how could you how do you have a perfect you can build perfect localizers are not that hard to build okay come on now with with wood lighter why don't ya wood lighter okay yeah but you use lighter right like use lidar build a perfect localizer building a perfect localizer without lidar it's gonna be it's gonna be hard you can get ten centimeters without liner you can get one centimeter with lidar maybe concern about the one or ten centimeter I'm concerned if every once in a while you're just way off yeah so this is why you have to carefully make sure you're always tracking your position you want to use light camera fusion but you can get the reliability of that system up to a hundred thousand miles and then you write some fallback condition where it's not that bad if you're way off right I think that you can get it to the point it's like özil D that you're you're never in a case where you're way off and you don't know it yeah okay so this is brilliant so that's the static static we can especially with lidar and good HD maps you can solve that problem easy no you just the static static very typical for you to say something's easy I got it it's not as challenging as the other ones okay well it's okay maybe it's obvious how to solve it the third one's the hardest well where do we get and a lot of people don't even think about the third one and even I see it as different from the second one so the second one is dynamic the second one is like say there's an obvious examples like a car stopped at a red light right you can't have that car in your map yeah because you don't know whether that car is gonna be there or not so you have to detect that car in real time and then you have to you know do the appropriate action right also that car is not a fixed object that car may move and you have to predict with that car will dim alright so this is the dynamic problem yeah do you have to deal with this um this involves again like you're gonna need models of other people's behavior do you are you including in that and I don't want to step on on the third one oh but if I are you including in that you're influenced and people I guess the third okay that's the moon we call it the counterfactual yeah I believe that I just talked to Judea pearl who's obsessed with counterfactuals oh yeah yeah so the static and the dynamic yeah our approach right now for lateral will scale completely to the static a dynamic the counterfactual the only way I have to do it yet they don't give you thing that I want to do once we have all these cars is I want to do reinforcement learning on the world I'm always gonna turn the exploiter up to max I'm not gonna have them explore but the only real way to get at the counterfactual is to do reinforcement learning because the other agents are humans so that's fascinating that you break you down like that I agree completely I've set my life thinking about this beautiful they're so and part of it because you're slightly insane because not my life just the last four years no no you have like some some nonzero percent of your brain has a madman in it which that's a really good feature but there's a safety component to it that I think when this sort of counterfactuals and so on that would just freak people out how do you even start to think about just in general I mean you've you've had some friction with Nitza and so on I am frankly exhausted by safety engineers the the prioritization on safety over innovation to a degree where it kills in my view kills safety in the long term so the counterfactual thing they just just actually exploring this world of how do you interact with dynamic objects and so on how do you how do you think about safety you can do reinforcement learning without ever exploring and I said that like so you can think about you're in like a reinforcement learning it's usually called like a temperature parameter and your temperature parameter is how often you deviate from the Arg max I could always set that to zero and still learn and I feel that you'd always want that set to zero on your actual system got you but the problem is you first don't know very much and so you're going to make mistakes so the learning the exploration happens to ready yeah but okay so the consequences of a mistake yeah open pilot and autopilot are making mistakes left and right yeah we have we have we have 700 daily active users a thousand weekly active users open pilot makes tens of thousands of mistakes a week these mistakes have zero consequences these mistakes are oh it I wanted to take this exit and it went straight so I'm just gonna carefully touch the wheel humans the humans catch them and the human disengagement is labeling that reinforcement learning in a completely consequence-free way so driver monitoring is the way you ensure they keep yes they keep paying attention how is your messaging say I gave you a billion dollars you would be scaling and now oh my fact it's guy couldn't scale with any amount of money I'd raise money if I could if I had way to scale yeah you're not focusing I don't know I don't know how to do Oh like I guess I could sell it to more people but I want to make the system better better I don't know I mean but what's the messaging here I got a chance to talk to you on and and he he basically said that the human factor doesn't matter you know the human doesn't matter because the system will perform there would be sort of a sorry to use the term but like a singular like a point where it gets just much better and so the human it won't won't really matter but it seems like that human caching the system when it gets into trouble is like the thing which will make something like reinforcement learning work so how do you how do you think messaging for Tesla for you should chant for the industry in general should change I think my messaging is pretty clear at least like our messaging wasn't that clear in the beginning and I do kind of fault myself for that we are proud right now to be a level 2 system we are proud to be level 2 if we talk about level 4 it's not what the current hardware it's not gonna be just a magical OTA upgrade it's gonna be new hardware it's gonna be very carefully thought-out right now we are proud to be level 2 and we have a rigorous safety model I mean not like like okay rigorous who knows what that means but we at least have a safety model and we make it explicit is in safety MD and open pilot and it says seriously though safety dot MD Android so well this is this is the safety model and I like to have conversations like if like you know sometimes people will come to you and they're like your systems not safe okay have you read my safety Doc's would you like to have an intelligent conversation about this and the answer is always no they just like scream about it runs Python okay what so you're saying that that because pythons not real-time Python not being real-time never causes disengagement disengagement SAR caused by you know the model is QM but safety dad MD says the following first and foremost the driver must be paying attention at all times I don't can I do I still consider the software to be alpha software until we can actually enforce that statement but I feel it's very well communicated to our users two more things one is the user must be able to easily take control of the vehicle at all times mm-hmm so if you step on the gas or brake with open pilot it gives full manual control back to the user or press the cancel button step 2 the car will never react so quickly we define so quickly to be about one second that you can't react in time and we do this by enforcing torque limits braking limits and acceleration limits so we have um like our torque limits way lower than Tesla's this is another potential if I could tweak autopilot I would lower their torque limit or would a driver monitoring um because autopilot can jerk the wheel hard yeah open pilot can it's we we limit um and all this code is open source readable and I believe now it's all misery C compliant misra is like the automotive coding standard um at first I you know I've come to respect I've been reading like the standards lately and I've come to respect them they're actually written by very smart people yeah they're brilliant people actually they have a lot of experience there's sometimes a little too cautious but in this case it pays off miss was written by like computer scientists and you tell them as a language they use you can tell by the language they use they talk about like whether certain conditions in misra are decidable or undecidable you mean like the halting problem and yes well all right you've earned my respect I will tell you carefully what you have to say and we want to make our code compliant with that all right so you're proud level two and reform so you were the founder and I think CEO of comm AI then you were the head of research what the heck are you know what's your connection to come AI the president but I'm one of those like unelect unelected presidents of like like a small dictatorship country not one of those like elected presidents oh so you're like Putin when he was like yeah I got sure so there's uh what's the governance structure what's the what's the future of commie I finance I mean as a business do you want you just focused on getting things right now making some small amount of money and mean to and then one that works it works in each scale our burn rate is about 200k a month and our revenue is about 100k a month so we need to 4x our revenue but uh we haven't like tried very hard at that yet and the revenue is basically selling stuff online yeah we sell stuff shopped a comment at AI is there other well okay so you you'll have to figure out that's our that's our only see but to me that's like respectable revenues yeah we make it by selling products to consumers we're honest and transparent about what they are most actually level for companies right because you could easily start blowing up like smoke like over selling the hype and feeding into getting some fundraisers oh you're the guy you're genius because you hacked the iPhone oh I hate that I hate that yeah I can trade my social capital for more money yeah I did it once I almost regret it doing the first of it well on a small tangent what's your you seem to not like Fame and yet you're also drawn to fame what were you on we're on you where are you on that currently have you had some introspection some soul-searching yeah I actually I've come to a pretty stable position on that like after the first time I realized that I don't want attention from the masses I want attention from people who I respect who you respect I can give a list of people so are these like Elon must have characters yeah well actually you know what I'll make it more broad than that I won't make it about a person I respect skill I respect people who have skills right and I would like to like be I'm not gonna say famous but be like known among more people who have like real skills who in cars doers do you think have skill not do you respect Oh Kyle vote has skill a lot of people away mo have skill and I respect them I I respect them as engineers like I can think I mean I think about all the times in my life where I've been like dead set on approaches and they turn out to be wrong so I mean this might I might be wrong I accept that I accept that there's a decent chance that I'm I'm wrong and actually I mean having talked to Chris Urmson sterling anderson i those those guys I mean I deeply respect Chris I just admire the guy he's legit can you drive a car through the desert when everybody thinks it's impossible that is that's legit and then I also really respect the people who are like writing the infrastructure of the world like the linus torvalds and the chris lab they're doing the real work I know they're doing the real work this every dog that Chris Ladin you realize especially when they're humble it's like you realize oh you guys were just using your oh yeah all the hard work they did him that's incredible what do you think mr. Anthony lowendahl ski what do you he's a he's another mad genius sharp guy oh yeah what do you think he might long-term become a competitor Oh tu cama well so I think that he has the other right approach I think that right now there's two right approaches one is what we're doing and one is what he's doing can you describe I think it's called pronto a certain you thing did do you know what what the approaches actually don't know embark is also doing the same sort of thing the idea is almost that you want to so if you're I can't partner with Honda and Toyota Honda and Toyota are uh like four hundred thousand person companies it's not even a company at that point like I don't think of it like I don't personify it I think of it like an object but a trucker drives for a fleet maybe that has like some truckers are independent some truckers Drive for fleets with a hundred trucks there are tons of independent trucking companies out there start a trucking company and drive your costs down or figure out how to drive down the cost of trucking another company that I really respect is uh not oh I should I respect their business model no auto sells a driver monitoring camera and they sell it to fleet owners if I that's right if I owned a fleet of cars and I could pay you know 40 bucks a month to monitor my employees this is gonna like reduces accidents 18% yeah it's it's so like that in the space that is like the business model that I like most respect is there creating value today yeah which is uh that's a huge one is how do we create value today with some of this then the link keeping things huge and it sounds like you're creeping in or full steam ahead on the driver monitoring - yeah which I think actually were the short-term value if you can get right I still I'm not a huge fan of the statement that everything is to have driver monitoring but will I agree with that completely but I'm that statement usually misses the point that to get the experience of it right is not trivial oh no not at all in fact like so right now we have I think the time out depends on speed of the car but we want to depend on like the scenes day if you're on like an empty Highway it's very different if you don't pay attention then if light you're like coming up to a traffic light and long-term it should probably learn from from the driver because that's to do I watched a lot of video we've built a smartphone detector just to analyze how people are using smartphones and people are using it very differently and there's a it's a texting styles there's videos yeah like I got billions of miles of people driving cars in this moment I spent a large fraction of my time just watching videos because it's never fails to to learn like it never I've never failed from a video watching session to learn something I didn't know before fact I usually like when I eat lunch I'll sit especially when the weather is good and just watch pedestrians with an eye to understand like from a computer vision I just to see can this model can you predict what are the decisions made and there's so many things that we don't understand this is what I mean about the state vector yeah it's I'm trying to always think like Gamma understanding in my human brain how do we convert that into how hard is the learning problem here I guess is the fundamental question so something that from a hacking perspective this is always comes up especially with folks well first the most popular question is the trolley problem right so that's not a sort of a serious problem there are some ethical questions I think that arise maybe will you want to met you or do you think there's any ethical serious ethical questions that we have a solution to the trolley problem Akane aye well so there is actually an alert in our code ethical dilemma detected it's not triggered yeah we don't we don't how you have to detect the ethical dilemmas but we're a level two system so we're going to disengage and leave that decision to the human you're such a troll hey no but the trolley problem deserves to be trolled yeah that's a beautiful answer actually I know I gave it to someone who was like sometimes people ask like you asked about the trolley problems like you can have a kind of discussion about it like boo you get someone who's like really like earnest about it because it's the kind of thing where if you ask a bunch of people in an office whether we should use a sequal stack or no sequel stack if they're not that technical they have no opinion but if you ask them what color they want to paint the office everyone has an opinion on that and that's why the trolley problem is that's it I mean it's a beautiful answer yeah we're able to detect the problem and were able to pass it on to the human yeah I've never never heard anyone say it nice escape route okay but proud level - I'm proud level - I love it so the other thing that people cope you know have some concern about with AI in general is hacking so how hard is it do you think to hack a nataas vehicle either through physical access or through the more sort of popular now these adversarial examples on the sensors be adversarial examples one you want to see some adversarial examples that affect humans hmm right oh well there used to be a stop sign here but I put a black bag over the stop sign and then people ran it all right adversarial yeah right like like like there's tons of human adversarial examples - um the question in general about like security if you saw something something just came out today I'm like there are always such high P headlines about like how navigate on autopilot was fooled by a GPS spoof to take an exit right at least that's all they could do was take an exit if your car is relying on GPS in order to have a safe driving policy they're doing something if you're relying and this is why v2v is such a terrible idea v2v now relies on both parties getting communication right this is not even so I think of safety security is like a special case of safety right safety is like we put a little you know piece of caution tape around the hole so that people won't walk into it by accident security is I put a 10 foot fence around the hole so you actually physically cannot climb into it with barbed wire on the top and stuff right so like if you're designing systems that are like unreliable they're definitely not secure your car should always do something safe using its local sensors and then the local sensor should be hardwired and then could somebody hack into your can boss and turn your steering wheel on your brakes yes but they could do it before common AI too so let's think out of the box and some things so do you think teleoperation has a role in any of this so remotely stepping in and controlling the cars no I think that if safety if the safety operation by design requires a constant link to the cars I think it doesn't work so that's the same argument using for v2i VTV well there's a lot of non safety critical stuff you can do with v2 I like v2 I liked v2 I weigh more than V B because Vita I is is already like I already have internet in the car right there's a lot of great stuff you can do with v2 I um like for example you can well where I already have v2 Waze is V die right ways can route me around traffic jams that's a great example of v2 I mm-hmm and then okay the car automatically talks to that same service like improving the experience but it's not a fundamental fallback for safety know if any of your if any of your if any of your things that require wireless communication are more than qm like have a nozzle rating you should you previously said that life is work and then you don't do anything to relax so how do you think about hard work well what is it what do you think it takes to as great things you know there's a lot of people saying that there needs to be some balance you know you need to in order to accomplish great things you need to take some time off each of reflects and so on now and then some people are just insanely working burning the candle at both ends how do you think about that I think I was trolling in the Siraj interview when I said that off camera right before I spoke a little bit we'd like get out spot this is a joke right like I do nothing it relaxed look where I am I'm at a party right yeah that's true so no of course I I don't um what I say that life is work though I mean that like I think that what gives my life meaning is work I don't mean that every minute of the day you should be working I actually think this is not the best way to maximize results I think that if you're working 12 hours a day you should be working smarter and not harder well so it gives work gives you meaning for some people other source of meaning is personal relationships yeah like family and so on you've also in that interview of Sirach or does the the trolling mentioned that one of the things you look forward to in the future is AI girlfriends yes so at the topic that I'm all very much fascinated by not necessarily girlfriends but just forming a deep connection with AI what kind of system do you imagine when you say AI girlfriend whether you were trolling or not know that one I'm very serious about and I'm serious about that on both a shallow level and a deep level I think that VR brothels are coming soon and are gonna be really cool it's not cheating if it's a robot I see the slogan already but there's I don't know if you've watched it just watched the black mirror episode i watch the one year yeah yeah oh the the Ashley - one way da no where there's two friends were having sex with each other and mo in the VR game your game it's just two guys but yeah one of them was was a female and yeah there's another mind-blowing concept that in VR you don't have to be the form you can be to animals having sex weird I mean I'll see you I said the software Maps the nerve endings right yeah yeah they they sweep a lot of the fascinating really difficult technical challenges under the rock like assuming it's possible to do the mapping of the nerve endings then I wish yeah I saw that the way they did it with a little like stim unit on the head that'd be amazing so wanna know on a shallow level like you could set up like almost a brothel with like real dolls and oculus quests right some good software I think it vehicle novelty experience you know on a deeper like emotional level I mean yeah I would really like to fall in love with with with the machine do you see yourself having a long-term relationship of the kind monogamous relationship that we have now with a robot with a a AI system even not even just the robot so I think about maybe my ideal future when I was fifteen I read eliezer yudkowsky early writings mmm-hmm on the singularity and like that AI is going to surpass human intelligence massively he made some Moore's law based predictions that I mostly agree with and then I really struggled for the next couple years of my life like why should I even bother to learn anything it's all gonna be meaningless when the machines show up right maybe maybe when I was that young I was still a little bit more pure and really like clung to that and I'm like wow the machines ain't here yet you know and I seem to be pretty good at this stuff let's uh let's try my best you know like what's the worst that happens but the best possible future I see is me sort of merging with the Machine and the way that I personify this is in a long-term monogamous relationship with a machine oh you don't think there's room for another human in your life if you really truly merge with another machine I mean I see merging I see like the best interface to my brain is like the same relationship and to merge with an AI right does that merging feel like I see yeah I've seen couples who've been together for a long time and like I almost think of them as one person like couples who spend all their time together and that's that's how you're actually putting what does that merging actually looks like it's not just a nice channel like a lot of people imagine it's just an efficient link search link to Wikipedia or something I don't believe in that but it's more you're saying that there's the same kind of the same kind of relationship you have one other human that's a deep relationship is that's what merging looks like that's that's pretty uh I don't believe that link is possible um I think that that link so you're like oh me to download Wikipedia right to my brain yeah my reading speed is not limited by my eyes my reading speed is limited by my inner processing locally and to like bootstrap that sounds kind of unclear how to do it and horrify but if I am with somebody and I'll use a somebody who is making a super sophisticated model of me and then running simulations on that model I'm not gonna get into the question whether the simulations are conscious or not I don't really want to know what it's doing um but using those simulations to play out hypothetical futures for me deciding what things to say to me to guide me along a path and that's how I envision it so on that path to AI of superhuman level intelligence you've mentioned that you believe in the singularity that singularity is coming yeah again could be trolling could be not could be part I'm all trolling his truth in it I don't know what that means anymore what is the singularity yeah so that's that's really the question how many years do you think before the singularity what form do you think it will take does that mean fundamental shifts and capabilities of AI does it mean some other kind of ideas um maybe this is just my roots but so I can buy a human beings worth of compute for like a million bucks that I it's about one TPU pod v3 I want like I think they claim a hundred peda flops that's being generous I think humans are actually more like twenty so that's like five humans that's pretty good Google needs to sell their teepees um but I could buy I could buy I could buy GPUs I could buy a stack of like by 1080 tea eyes build data center full of them four million box I can get a human worth of compute but when you look at the total number of flops in the world when you look at human flops which goes up very very slowly with the population and machine flops which goes up exponentially but it's still nowhere near I think that's the key thing to talk about when the singularity happened when most flops in the world are silicon and not biological that's kind of the crossing point like they are now the dominant species on the planet and just looking at how technology is progressing when do you think that could possibly happen you think go to happen in your lifetime oh yeah definitely my lifetime I've done the math I like 2038 because it's the UNIX timestamp rollover yeah beautifully put so you've you said that the meaning of life has to win if you look five years into the future what does winning look like so [Music] hi there's a lot of I can go into like technical depth to what I mean by that to win um it may not mean I was criticized for that in the comments like doesn't this guy want to like save the penguins in Antarctica or like you know listen to what I'm saying I'm not talking about like I have a yacht or something I am an agent I am put into this world and I don't really know what my purpose is but if you're a reinforcement if you're if you're an intelligent agent and you're put into a world what is the ideal thing to do well the ideal thing mathematically you go back to like Schmitt Hoover theories about this is to build a compressive model of the world to build a maximally compressive to explore the world such that your exploration function maximizes the derivative of compression of the past mid Hoover has a paper about this and like I took that kind of as like a personal goal function so what I mean to win I mean like maybe maybe this is religious but like I think that in the future I might be given a real purpose or I may decide this purpose myself and then at that point now I know what the game is and I know how to win I think right now I'm still just trying to figure out what the game is but once I know so you have you have imperfect information you have a lot of uncertainty about the reward function and you're discovering it exactly the purpose is that's that's the better way to put it the purpose is to maximize it while you have it a lot of uncertainty around it and you're both reducing the uncertainty and maximizing at the same time yeah and so that's at the technical level what is the if you believe in the universal prior yeah what is the universal reward function that's the better way to put it so that when it's interesting I think I speak for everyone in saying that I wonder what that reward function is for you and I look forward to seeing that in five years in ten years I think a lot of people who do myself right and cheering you on man so I'm I'm a happy you exist and I wish you the best of luck thanks for talking today man thank you this is a lot of fun you

Channel: Lex Fridman

Views: 784,395

Keywords:

Id: iwcYp-XT7UI

Length: 119min 36sec (7176 seconds)

Published: Mon Aug 05 2019

Reddit Comments

[u/thisiswhatidonow]

Super interesting talk. Few Tesla related points:

• Elon offered George 12 million to emulate AP1 in 12 months (edit: would drop by 1 million every month it was not delivered)
• Tesla will win lvl5 race due to scale of data collection
• NoA lane change should vary in speed so it feels more natural
• Transition between manual driving and AP should be more fluid
• Tesla AP should have better driver monitoring
• lvl5 will not be reached in the next 2 years
• LIDAR is useless

[u/Tbrou16]

Lex Fridman on Joe Rogan was really good, too

[u/DTTD_Bo]

Such a good podcast. I really enjoy George Hotz. He speaks highly of Tesla and Elon

[u/ricksastro]

Awesome...I really loved his comment on using the wheel for both drivers monitoring and for disengagement. It drives me crazy, since my Honda had better integration of LKA. It basically never disengaged LKA. If I changed lanes manually, the LKA would pause and then auto re-engage when it could. Sooooo much more seamless and intuitive.

And his comments on NOA are just spot on IMO.

[u/[deleted]]

my god this interviewer. he's such a slow talker and he's so bad at summarizing what the interviewee is saying that it kills me. he gets the best guests but i want to pull my hair out watching him talk. same thing last time with elon musk. i'm sure the guy is super smart but man he sounds like an idiot.

[u/DeusExWars]

1:30 I guess the 3th problem mentioned, is what Tesla is doing with Project Dojo

Reinforced learning

[u/mellott124]

Why is he so angry looking?

[u/TechVelociraptor]

'I am an agent. I am put into this world and I don't really know what my purpose is. But if you're if you're an intelligent agent and you're put into a world what is the ideal thing to do? Well the ideal thing mathematically, you go back to like Schmidhuber's theories about this, is to build a compressive model of the world to build a maximally compressive... to explore the world such that your exploration function maximizes the derivative of compression of the past. Schmidhuber has a paper about this and like I took that kind of as like a personal goal function. So what I mean to win I mean like maybe... maybe this is religious but like I think that in the future I might be given a real purpose or I may decide this purpose myself and then at that point now I know what the game is and I know how to win I think right now I'm still just trying to figure out what the game is, bit once I know...'

I don't think he is trolling with this one. The paper he refers to may be this one.