Season 1 Ep.1 Andrej Karpathy on the visionary AI in Tesla's autonomous driving

[Music] today here with me is andre kerpafi andre is the director of artificial intelligence and autopilot vision at tesla he's one of the world's leading visionaries in deep learning and computer vision ever since his phd days at stanford back when deep learning was just starting to emerge as a viable technology ander has not only been a leading researcher but also leading educator from there he became one of the founding members of openai the ai research organization in san francisco initially largely funded by elon musk and from there at some point elon musk then recruited andre to head up autopilot efforts at tesla andre you and i used to spend so much time together for listeners andre and i used to work together at openai but we also used to see each other at all of the ai conferences academic working groups but now it's actually been quite a while since we've seen each other and so i'm excited to get to catch up not even sure when is the last time we saw each other maybe nerves maybe my wedding maybe covering offices do you remember i do it was just before the pandemic i think right before pandemic was always i've been at nerves in vancouver yeah i remember i remember the the burgers we went to eating as a live bar across the conference center yeah those were days um and of course i visited covariant and saw your offices and uh we had a chance to talk there which was a lot of fun i hope we can host you again sometime soon um now i actually spent most of yesterday watching all the videos listed on your website absolutely brilliant what you've achieved and how you explain it there now can you tell our listeners a little bit about how you went from slovakia age 15 to becoming tesla's ai director that's a quite all-encompassing question yeah it's a long story so as you mentioned my parents decided to immigrate to canada when i was 15. um and yeah i was not a very happy sort of person in slovakia i always wanted to leave slovak has not a not an incredibly ambitious place and i felt that um i was upper bounded in terms of what i can achieve there and so i always wanted to go to canada america and uh do something substantial and um so when they asked when they kind of hinted that we may be able to uh go to canada i was on board instantly um and then my sister was not quite as on board and you know everyone else in the family as well but i sort of worked to convince everyone make the move and yeah so we came to canada i started in high school i barely spoke any english i had to learn english i was very good at math luckily the slovakian curriculum is quite good at math and so i was able to sort of get into their good courses and go through high school join the university of toronto and kind of get into computer science and and so on uh it is a long story it is a long story but i think there are some really interesting parts to it for example somewhere i read that jeff hinton at toronto is the one who first showed you neural networks yeah yeah so in the university of toronto is when i first came into contact sort of with deep learning through jeff hinton's class and he taught the class and at the time this was a very very simple primordial deep learning so we were looking at little digits white on black three two four one and we're trying to recognize you know those digits and so by today's standards these were incredibly toy problems um and uh i was fascinated by the way that jeff hinton spoke about these algorithms um he kept making analogies to the human brain and um you know the way he talked about the neural net it was like in the mind of the network and he kept using these anthropomorphic descriptions for what was going on in the algorithm and i found that analogy fascinating and so that's kind of what pulled me in i would say into the entire area and then yeah i i audited the class um and i also went to some of the reading groups that he hosted with his students and that's basically when i became extremely interested in deep learning and have been in deep learning since now if we jump ahead a little bit i think the first time you really were very visible as somebody in uh deep learning was during your phd days at stanford when you were the one who were generating a lot of the research progress and educational content can you say a bit more about uh how do you get going on that at stanford yeah so i think you're alluding to cs231n the class that i ended up basically uh designing to a very large extent and then being the primary instructor for together with faith hey and it was really the first deep learning class at stanford and became extremely successful so in the first offering it was maybe 150 students or so and by the time i left it was 700 or so and so this became one of the biggest classes at stanford um and the way it came about i've always had a passion for teaching and even during my master's degree at university of british columbia ta in different classes was always the highlight of my experience i just love watching people learn new skills and then go out and do cool things with those skills and i feel like it's such a large lever over you know kind of impact it's indirect but i think it's a very large lever and so i was i was really um you know just very passionate about teaching in general and deep learning at the time was starting to have some of its first successes uh so in 2012 uh we had the first um convolutional neural network uh give a stunning performance on this imagenet benchmark in in computer vision and image classification and a lot of people suddenly paid a lot of attention to deep learning and i happened to be in a place where i understood the technology and i was very passionate about teaching and faith approached me and she pitched me on the idea that hey we could start a class and actually teach people about this and i instantly i instantly jumped at the idea i loved it i put my entire phd in research on hold it's not something you would typically want to do as a phd student because your primary output is research and i ended up not doing basically any research for maybe a year because i just focused on the class and its design but i felt like it was probably more impactful than getting a paper or two out to actually like do this class and do a good job of it and yeah so we came up with the curriculum and taught it and it was amazing and it was probably the highlight of my phd so andre i mean a lot of people already know this but some people probably don't i mean you're saying this class went from 150 to 700 students from year one to year two but the reality is much bigger than that of course i mean this class was put on youtube and there were hundreds of thousands of if not millions of people watching your lectures because it wasn't just the first stanford class it was the class that everybody was watching following along because deep playing was this new thing and it was the place you would go if you wanted to understand it you had to go to andre karpathy's class that that's where everybody went and so i think yeah i just want to clarify there's more than 700 people who were active in that class um yeah absolutely i did think it had a larger reach than i had anticipated at the time of course by a lot i still people come up to me randomly in conferences and even in like in a coffee shop and tell me that they've uh you know they saw my class and they really enjoyed it and you know i i've heard it many times but it always brings the same amount of pleasure and so i feel like that was a very good investment in time very proud of it when you go and check out one of andre's videos the most frequent comment is just two words my hero that's that's what you say because that you opened up a whole field for so many people that was hard to access and all of a sudden they could be part of it um now how would you explain that that moment that triggered your teaching the class you said imagenet happened in 2012. what happened there what was that all about yeah so as i mentioned when i first encountered deep learning in jeff hinton's class we were working with these tiny black and white images so these are tiny 28 by 28 images where you have like a single digit like a seven or six and we're trying to recognize like what it is and so deep learning which is this class of neural network neural networks approaches where you basically have neurons that are connected to each other with different strengths and you're trying to tune the strengths between the neurons so that they take your input in this case this image and try to you know the the neurons fire in the sequence and then the last neurons tell you which digit it is in the image and so it's a class of technology that was used in these super toy problems and when i entered uh stanford and i was in computer vision these approaches were not uh what you would use for computer vision applications so computer vision was a field that was mostly working with much larger higher resolution images so we're talking you know what one two three megapixel inputs so 1000 by 1000 images real normal big images and it was thought and there was a very different class of approaches to how you um you know attack computer vision problems in those settings and so at the time no one was using deep learning in computer vision deep learning was this branch of machine learning out there somewhere and it was very different not used and it was thought the conventional wisdom at the time was that these approaches would not scale to large resolution images and it was in 2012 that uh jeff hinton and his team um in the university of toronto uh ask kurchevsky and elia suskover published a paper showing that a scaled up version of the neural network uh really running running on special you know gpus in the computer a special type of processor that is um very that is very good at running these kinds of computations that make up the neural network that when you scale up these networks so it's not just a little baby network with a couple thousand neurons but it's actually a much bigger network with several hundred thousand of them connected with like millions of uh weights and synapses that actually these can do incredibly well even on normal sized images and achieve extremely good performance compared to what was available in computer vision at the time and this turned the entire field upside down it was a massive tectonic change in computer vision so if you visited a computer vision conference in 2012 there would be basically like one or two papers on deep learning but if you visit it five years later it would be one or two papers that are not deep learning it was a complete upheaval and today of course he would not dream of doing computer vision without deep learning so i happened to be at the right place at the right time with interest in education and uh stars sort of aligned in the way that i was able to explain the material and hopefully empower a lot of people to use this powerful technology for computer vision problems in the world now you absolutely did and i'm curious if you had to describe just deep learning let's put aside image in that moment for just deep learning itself how would you describe deep learning to i don't know your parents or you know uncle or something who doesn't work in in the space let's use a specific example because i think it's useful so let's let's talk about image recognition right so we have images and they are just um images are made up to computer of a large number of pixels and each pixel just tells you the amount of brightness in the red green and blue channel at that point and so you have a large array of numbers and you have to go from that to hey it's a cat or a dog and typical conventional software is written by a person programmer writing a series of instructions to go from the input to the output so in this case you want someone to write a program for how do you combine these millions of pixel values into like is it a cat or a dog turns out no one can write this program it's a very complicated program because there's a huge amount of variability in what a cat or doc can look like in different brightness conditions arrangements poses occlusions basically no one can write this program so deep learning is a different class of programming in my mind where no one is explicitly writing the algorithm for this recognition problem instead we are structuring the entire process slightly differently so in particular we arrange a large data set of uh possible images and um the desired labels that should come out from the algorithm so hey when you get this input this is a cat when you get this output this should be a dog and so on so we're kind of stipulating what is the desired behavior on a high level we're not talking about what is the algorithm we're measuring the performance of some algorithm and then you need some and then roughly what we do is we lay out a neural network which is these um it's it's a bunch of neurons connected to each other with some strengths and you you feed them images and they predict what's in them and the problem now is reduced because um you're just trying to find the setting of these synaptic strengths between the neurons so that the outcomes are what you want and so as an example the 2012 imagenet model which was roughly 60 million parameters so the weights of the neural network were really 60 million knobs and those knobs can be arbitrary values and how do you set the 60 million weights so that the network gives you the correct predictions and so deep learning is is a class of is a way of of training this neural network and finding a good setting of these 60 million numbers um and so roughly uh the neural network sort of looks at the image gives you a prediction and then you measure the error it's like okay you said this is a cat but actually this is a dog and then and there's a mathematical procedure for tuning uh the strengths so that the neural network adapts itself to agree with you and so deep learning is is basically a different software programming paradigm where we specify what we want and then we use sort of mathematics and algorithms to tune the system to give you what you want and there's some design that goes into the neural network architecture and how do you wire everything up but then there's also a huge amount of design and effort spent on the data sets themselves and curating them and you know because those data sets are now your constraints on the behavior that you are asking from the system so it's a very different way of approaching problems that was not there before everything used to be written by person now we just write the specification and we write a rough layout of the algorithm but it's a it's what i refer to as fill in the blanks programming because we sort of lay out an architecture and a rough layout of the net but there's a huge amount of blanks which are the weights and the knobs and those are set now during the training of this network so that's i think the bird's eye view of how this is different i like the way you explain that and and it's of course no coincidence that you're heading up self-driving at tesla and that you're one of the world leading experts in this exact discipline of deep learning there's got to be a strong connection there i want to i want to go a little bit towards the tesla side of things what was the moment you decided to join tesla how did that come about yeah so um after my phd at stanford i went to be a research scientist one of the founding members at openai which is where we overlapped briefly as well and i spent about almost two years at openai and by the time of two years at open ai i have been doing research for about a decade so my master's degree my phd and then open ai and so i spent about a decade reading and writing papers and working on you know training neural networks but in an academic setting mostly and so i was definitely getting a little bit restless at that time because i felt like these these algorithms are extremely powerful and uh can can really move the needle um on some very incredibly important problems in society and um i wanted to take a more active role in doing that and so i was getting a bit restless i was looking at different opportunities and say startups and things like that and then one thing that kind of happened on the side is because openai was at the time under the umbrella of elon organizations a few times we were interacting with people at tesla and i was kind of consulting a little bit for some of the problems in the autopilot and um i kind of realized that they were dealing with fundamentally a deep learning computer vision problem and this was the fundamental constraint to whether or not this product would work and so i was kind of like intrigued by that but it was just a few consulting opportunities in here and there i sort of spoke to the team but at this time when i was getting really restless to applied technology in the industry actually elon reached out and he asked me hey you've been like sort of consulting for for the team um do you actually want to join in and lead the computer vision team and the ai team here and um help get this car to drive and so he caught me at a very kind of correct time when i was really getting restless and i felt like this is perfect and i think i can do this i think i have the skills to contribute here this is an incredibly impactful opportunity and i love the company and of course i love elon and everything that he's doing and so i would say that again sort of it was a moment where stars aligned for me and i felt very strongly that this is the right thing to do at this time and so i left open ai and i've been at tesla for the last four years or so almost uh so yeah it's been some time yeah i've been there for four years and kind of finally i remember this moment where you were just about to leave open the eye and a bunch of us are chatting about your plans and you're joking uh but you're also half serious and and you're saying well you know this is a job that on average uh people last six months and then you want to take time for somebody else to take over and here you are an average six months leading up to when you started and now almost four years and knowing this i think knock on wood um of how this is going this is just amazing uh yeah that that's accurate i was aware of the average tenure uh at tesla especially when you're working on very important projects very close to elon and so um yeah i was very much aware of this so when i made your transition to tesla for example i did not give up my apartment in san francisco because i was just kind of like not you know really hedging my bets on what's going to happen in the next few months but uh yeah here i am for four years later still here yeah amazing i'm curious um if you look at the tesla mission statement it's about transition to renewable energy right um which i'd like first sight doesn't seem to directly tie into self-driving being kind of part of wanting to transition to renewable energy so can you say a bit about how self-driving and renewable energy play together yeah i think it's it's a good question i think broadly elon sort of has a number of companies and a number of bets around just a higher level goal of you know making the future good increase the probability of the future being good um and you know there's there's many aspects of the to that of course and he's vote he's focused the tesla mission around um you know accelerating the transition to sustainable energy fundamentally a large piece of this is getting people to transition to electric vehicles and we need to manufacture them at scale and we want them to to look like the future and so the entire product itself uh sort of looks like the future it's a very clean design and uh you want to be inspired by progress in society and and that things are developing in a positive direction and so the car looks much more futuristic and i think a big part of that also is that the car becomes uh sort of um you know it just becomes something magical in your life that can take you around in this beautiful future and so i think autonomy really is um part of just a broader vision to to this future that we want to be part of where we are driving uh electric vehicles with very little footprint and uh the society is sort of automated to a large extent and there's a huge amount of problems of course also around transportation and putting people in the loop with the amount of accidents that they get into and also with the fact that you don't want people to be really driving these cars because human brain is capable of so many beautiful things uh so why should you solve the line following problem you know that that is not a good use of the brain uh bit harder than have to build it though isn't it so not only is it unsafe to drive these cars it's also just you want the brains to be doing something different um and so we have the technology to to address this um so that's why yeah we're working on it that's really intriguing that the way i'm internalizing this is that if you want people to transition off something they like people right like their existing cars you gotta you can't just tell them let go of your existing cars and you know stop using them you got to show them something even shinier something even more exciting that in the process also gets them onto renewables yeah we want people to transition to electric vehicles that are also very competent in the world and transport you around before we dive into the technology itself for self-driving got got another question at the higher level which is how is it working with elon musk i mean he he might well be the most famous person in the world at this point and you are actually working with him what is that like well he's obviously a very incredible person in many ways i'm still trying to really map out his superpowers he has incredibly well developed intuition i would say in many aspects where uh he makes the right judgment calls sometimes in what i perceive to be a lack of information uh like because he's not fully in detail of all the things um but yet his judgment is extremely good um and so i i still haven't fully sort of understood how that happens he has a way of taking a very complex system and simplifying it to just like the fundamentals and the really the first principal components of what really matters about the system and then making uh statements about about those and so it's a very different way of thinking that i find kind of fascinating by default for example um sometimes i get sort of overwhelmed by the system i feel like i need to know the system in its full detail to make the correct decisions but that's not how he operates he somehow has a way to distill the system into a much simpler system in which he operates and so i think i've learned a lot about just how to approach problems and uh you know he's a double-edged sword because um in terms of working with him right because he wants the future yesterday and uh you know he uh he will push people and he will inject a lot of energy and he wants it to happen quickly and you have to be of a certain i think uh attitude to really tolerate that over long periods of time but he surrounds himself with people who get energy out of that and they also want the future to happen quicker and those people really thrive at tesla and so i happen to also i think be like that and so i don't personally mind it i actually kind of thrive on it and i love the energy of of getting this to work faster and uh you know making a difference and having this impact and so i really enjoy working with him because he has a way of injecting energy into the system driving momentum and he has incredibly good good and developed judgment and so yeah i overall just really really enjoyed working with him sounds wonderful would you say you talk with him pretty much every week or whatever that's right so we have autopilot meetings that range from a week to multiple times a week depending on uh you know just how much scrutiny is being put on the autopilot maybe right in front of releases we would have uh more than a week and multiple times in the history of the team it's been every single day um and uh so yeah on any of those frequencies depending on what's happening that's so exciting wow now if we think about self-driving cars it's probably the kind of most tangible ai concept for the public because so many people have cars and it's how their car is going to change because of ai right and certainly one of the most written about aspects of ai research and application in the press but not everybody really realizes how driverless cars and ai are connected what is the backstory there how long have people been working on self-driving cars and what is the ai role what is what is happening under the hood yeah so people have of course been thinking about cars that drive themselves for a very long time some things are very easy to imagine but very difficult to execute on like driverless cars some things are not like that um so for example a cryptocurrency in bitcoin is hard to sort of come up with so you won't see something like that maybe featured in as much sci-fi but driverless cars are something that people have been dreaming about for a very long time and working on for a long time um and i think fundamentally what makes it hard is right that you have to deal with a huge amount of variability of what the world looks like and it's roughly it's basically true that for ai and technology as it is today the degree of difficulty is proportional to the degree of variability you're going to encounter in the application so the more scenarios you have to deal with the harder it will be for the technology and that's what makes this hard for self-driving cars as well is that environments out there are quite variable maybe on the highway you're just dealing with lane following but once you get off the highway into city streets for in san francisco and so on the the amount of things you can encounter is very large and designing to it is incredibly difficult and that's where all the action is you hit upon variability right that that's that's making it so hard can you dig a little deeper why this variability make it hard so like i mentioned you want you are giving like when you're creating these deep learning systems you are giving them some kind of a specification for how they should act in different environments in different cases so hey this is a cat this is a dog and um the network starts from scratch it's not like your human brain that is born into a three-dimensional physical reality where you sort of understand a lot of objects and you come with all these all this built-in hardware but then also incredibly powerful learning algorithms so you can understand objects object permanence and how the world works these neural networks um they are made up of neurons like your brain but they function they actually it's a very it's not an exactly correct analogy and it's misleading these neural networks again it's better to think of them as a mathematical function with a lot of free parameters 60 million knobs that must be set to get the correct behavior and in the beginning the setting of these knobs is completely random so the neural net is implementing a completely random function it's doing completely random things and it's starting basically from scratch and so you have to tell it what to do in every situation and the more situations you have the more you're going to have to give it in order for it to do the right thing in all the cases so andre when the neural network starts from scratch and you put that neural network on a tesla what what would happen if it drives at tesla well you'll get random behavior when it's from scratch it'll be completely random behavior got it so it starts not knowing what to do so you probably don't put those on on the cars actually no you don't want to do that and so how when you deal with all this variability and you want this neural network to internalize that variability um can you say more about what i mean what what makes the neural network internalize that variability what's what's the solution to that it so it looks like um we do it through roughly almost brute force ways right now so if i want the neural network to function in millions of situations i need to plug in millions of examples and or something on that order so the neural networks do show some ability to sort of interpolate between the examples you've given them they are they're not as good at extrapolating but as long as you sort of cover the space of possibility and tell the neural network what to do in those different scenarios they have some ability to interpolate between examples but it's it's limited and so if you really have a massive amount of variability that you want the system to perform well on uh you actually have to cover that space uh to a large extent so how how do you how do you get the data to cover that space as i mentioned in this new programming paradigm there's a bit of designing the neural network and the neurons and so on but a massive amount of work is on curating uh these data sets and um fundamentally you you roughly start with a data set um you start with some data set and you train your neural network and then you measure its performance and you look at where it works and where it does not work and the fundamentally the way you're iterating on this neural network to get it to work is you need to find a lot of examples where it does not do what you want to do and then you need to get those situations and you need to label correctly what should have happened in those situations where the correct label would have been in all those cases and then you need to put those into the training set for the neural network and so the neural network is now trained on the previous data set but also on a data set of where it failed before but now has the correct label and this improves some situations again and then you have to again look at where it's failing now and the faster you can spin this loop of just iterating and curating your data set um the the better this neural network will become and luckily we are in a position with these deep neural networks that as long as the data set is um as long as the data says improving there's no real upper bound on the performance of the network um with if you have enough computation available for it and a large enough data set it will find the correct sort of solution to making your labels work so most of the engineering is on the data set and primarily it comes from sourcing examples where you're not working yet and sourcing examples where it's not working yet is is that when i drive my tesla is am i sourcing those examples and how does that work uh yeah exactly so um it's a great question a lot of what i do of course at work is just curating these data sets as i mentioned that's where all the engineering now is it's not people writing algorithms it's people collecting data sets and uh so you know for example there's lots of things we want to know about the scene right so we want to know where the lines are where the edges are where the traffic lights are where the other cars are whether or not the car door is open on the car if the left blinker is on a huge amount of things so roughly we have maybe say 30 top level tasks but a lot of those tasks have many sub tasks like for a car you may want to know a lot of attributes about it what kind of a vehicle is it you know is the car door open and so on so you end up with a huge amount of predictions that your neural network has to make about the world and um there are and now these networks are deployed in people's cars and they're running and making predictions and then we have to come up with lots of ways to source inaccuracies and there's many ways by which we do that um maybe one very simple example is if you intervene because the autopilot did not do something correct typically when you intervene in a large number of cases that has to do with an incorrect prediction from the network so an intervention is a trigger and we collect some of those images and then we look at them and we look at whether or not the predictions were correct and how are they wrong and that helps us triage should this example go into what labeling project and where should it end up in in what data set and with what label and that's how we sort of iterate on the system but there's many triggers that are active at any point in time as one more example if you have a detection of say a stop sign or something like that if the so you have a bounding box that the computer is putting around the stop sign and if the stop sign detection uh flickers for example so it's there and then the network says oh it's not a stop time oh wait it is stop sign when you see this disagreement with itself over time that also is typically an extremely good source of data so flicker and temporally consistent predictions or for example disagreements with the map so we think there's a stop sign but the map says that there isn't one so there's lots of different ways by which we gather examples where the network is mispredicting and for us it's an exercise of how quickly can you enter those examples into a training set and that's a huge portion of what the team is doing when i try to think about the data you're feeding into the system how much data is that i mean are we thinking thousands of images millions what magnitude are we talking about here yep so we're talking about millions of images easily it's on that order so millions of images that are annotated with all kinds of information the neural network should extract yes automatically in the future from similar images yep that's amazing now one of the recurring themes it seems in deep learning is um large data but also um large compute let's say you want to train the autopilot from all that data you say okay i'm going to retrain it push all the data through the new network and train it how much compute does it take how long does it take to train an autopilot yeah so what you're getting what you're getting at is these neural networks are quite expensive to train so we start with millions of images and a typical neural network if you want to typically what you will see in the industry is most networks train roughly on the order of two to three weeks of time because two to three weeks is actually more of a psychological reason for that is because that's the amount of time that a person is willing to wait for for the network to converge and to measure its performance um so but yeah they have to look at a lot of examples they have to make a lot of predictions and they have to be corrected on the predictions they're making during the training loop and this takes a long time and as you are scaling up the amount of compute available you can afford to um to use a bigger network and a bigger network will almost always work better uh but it needs more training time and so we're in a place where we are and this is a beautiful place to be by the way we are not constrained by human ingenuity and algorithms as used to be the case in computer vision because we had a class of approaches that leveled off and then we were the constraint but now we human ingenuity is not a constraint the constraint is the size of the data set and the amount of compute that you have available to you the algorithm now is known everyone knows the same algorithms and we just need to run them at scale and we're getting benefits for free just by scaling up the network making a bigger network and making a bigger data set so it's a beautiful place to be because you have a recipe a template by which you can make progress and you're just bounded by um very concrete tangible things that you can improve like the size of your training cluster and things like that where here said that the algorithms understood that that's true of course it still requires some true expertise in the space to to understand those algorithms but you're right they're not secret i i hear part of what you're saying it seems like you are spending a lot of your time on the data itself and a lot less on changing the algorithms what does that look like i mean i imagine you have a large team that helps with the data and so forth like what does that look like organizationally yeah and i think like to your point briefly um it's a good observation that the algorithms it's not fair to say that they're fully figured out and known it's i would say it's more true in some domains than others like in computer vision i think we we have a class of algorithms that we're pretty happy with for the simplest image recognition problems in many cases for example you're dealing with robots doing pick and plays and things like that i would say algorithms are absolutely much less known um and so different domains will will have will have different maturity of the technology available um and i also want to say that it's not the case that we spend zero time on algorithms it's more like we spend 25 percent of the time not 100 of the time and the only reason i typically point it out and stress that is because uh typically people coming from say academia have an expectation so in academia when you're working with neural networks typically your data set is fixed because we have certain benchmarks that we're interested in driving up so your data set is fixed like say the imagenet and your task is to iterate on the algorithm uh and then neural network design and layout to improve the numbers um and so everyone's 100 of the time on the neural network itself the structure the loss function and all the pieces of that and data says fixed and my reaction is to it is strong only because when you're in the industry you will iterate a lot on the data set as well so that's not to say that the algorithm design and modeling um is not there it's just uh it's the second order effect of what you would be doing um it's sort of the second term in the equation um and as i said it also varies per area so i would say um in robotics it's much less certain how to lay out the problem um how you structure it how you arrange it what is the data set how do you um what what labels are you collecting at what level of abstraction huge huge design space and not obvious what works yet uh but i would say that's less the case in just simple image recognition well i i like i like that you expanded them up on that the thing i'm actually curious about is how this relates to this term you coined a little while ago software 2.0 because it seems very related yeah yeah exactly so um software 2.0 was uh kind of like a blog post i published um a few years ago and it was just making the point that you know of course we have a lot of uh software that's driving large parts of society and automation in um in a space of information and so on and um a lot of the software right now is written by people uh so you know banking systems and you know internet search and things like that everything is sort of algorithms developed by people in principle understood and uh orchestrated in a certain way and [Music] it seems to me basically that with progress and deep learning um it you can sort of think of that neural network as a piece of software um but the software was not written by person this per the software was written by an optimization and so it's kind of like a new programming paradigm that we are not directly writing the algorithm we are programming the data sets and the algorithm really is an outcome of this training process or this compilation um as which would be sort of the equivalent in typical software so you would take your source code and you would compile it and get a binary so here the source code are the data sets the compilation is the training of the neural org and your binary is the final neural net the weights and to me what's happening in society right now is that we are well number one a lot of software that we couldn't have written before is now possible to write like image recognition systems but also a lot of system that a lot of software that used to be written in by instruction software 1.0 style can now be ported over to this more powerful paradigm uh to software 3.0 and the programming sort of looks different and the reason i wrote that post is that it's a little bit of a call to arms to all the engineers in that over the la we've been programming in the software 1.0 paradigm for four or five decades and we have a huge amount of infrastructure to help us program in this paradigm so we have ides that help you write code they point out bugs they do syntax highlighting there's a huge amount of software infrastructure we've built to help us program but this is not yet the case in this new programming paradigm so we have to develop completely new tools around data set curation monitoring the deployment of these neural networks the iteration the fine tuning everything that goes into programming this new paradigm is an uncharted territory and the tools that we have to iterate on these data sets are extremely primordial and i think can be improved a lot and so really the post was about pointing out that this is not just some kind of a classifier in machine learning this is actually a restructuring of how we write software and people have to take it seriously and we have to borrow a lot of what we've done with software 1.0 infrastructure and that helped us program and we have to port equivalents into working with neural nets because a lot of software will start to look like weights in the neural net it won't be cpos plus or python or whatnot and what do you say at this point when you talk about this neural nets effectively being the program to build a self-driving car is it just a neural net that's been trained with a lot of data or are there still other components yeah yeah that's a really good question so in the car uh there is there are both so images enter in the beginning right and we have pixels of an image telling us fundamentally what's out there in the world and then uh neural networks are doing some portion of the recognition so they're telling you hey there's a stop sign person etc but you can't just directly drive on person stop sign etc you have to actually write some logic around how do you take those intermediate uh sort of representations and predictions and you want to avoid the pedestrian and you want to stop at the stop sign and so there's still a lot of software 1.0 code sitting on top of the neural net and that code is basically reacting to the predictions so that it speeds up slows down turns the wheel to stay in the lane line markings and so on what i have seen in the history of the team since i've joined in four years ago is that um and this is also why i think is that really we've been porting a lot of the functionality from the software 1.0 land into the neural network and so originally the neural networks would only make predictions for example for a single image and they would tell you okay there's a there's a piece of a road edge but we actually don't just have a single image we have eight images right uh coming from eight different cameras that are surround in the vehicle so every image independently predicts little pieces of road edges and curves but there needs to be something above it that stitches it up into a three-dimensional sort of bird's-eye view of what's happening around the vehicle and that was all done in software developed by people so you take road edges from here you project them out go digest from all the cameras project them out stitch them up across boundaries and then over time you need to also stitch them up and track them and make it serve temporally continuous and all that was written by people and what we've done since then is is the neural network has engulfed a lot of the pieces of the engineering so the neural networks that are in the car today will not make a prediction per image um they will make prediction directly in the bird's eye view so they will say okay i've seen these eight images and from that i can see that the road edges are this way around the car and also i've seen the images over time and i've done the tracking and having accumulated information from all those frames here's actually what the world looks like around you and so pieces of the software 1.0 code are being engulfed by the neural net and it's taking on more and more responsibility in the stack and maybe at the end of the day this can all just be a neural net so maybe there's a very little room for engineering maybe the images just come in and what comes out is just what you really want which is the steering and the acceleration easily said hard hard to do but that is the final conclusion i would say of of this kind of a transition and there's very little software written by people it's just a neural net does the whole thing um yeah that's the holy grail i would say we are dropping new interviews every week so subscribe to the robot brains on whichever platform you listen to your podcasts now when when people think about neural nets often part of the reaction is um at least in in the early days was it's hard to understand what they do and and here you are putting a neural net as part of the decision-making system for driving people which is of course i mean a very uh risky thing if the autopilot makes mistakes right so how do you build confidence in the system and how i imagine you have early rollouts sometimes in your own how do you decide you're willing to try it out you know so maybe maybe directly engineered code is in is in charge of a lot of the um the stack but i think it gives a full sense of understanding of the entire system because ultimately this can be hundreds of thousands of lines of code so yes you can analyze individual functions but this is a very complex dynamical system and i think you may have a false impression that you actually understand the system um even though you understand like the individual components i would say really what it comes down to is you want a very robust process for really testing the hole and subjecting it to a huge amount of evaluation maybe both in for all the individual components making sure that okay the detection itself works and all the pieces of the neural network individually by themselves but then also end-to-end integration tests and you just want to to test the system and you want to do this whether or not a neural net is in charge and you want to subject it to say a huge amount of simulation to make sure it's working as as um expected and also of course through driving and so we have a large qa team that drives the car and uh you know verifies that everything is working you know as well as possible um and so we have a number of mechanisms by which we test these systems another one that's big for us is shadow mode releases so you can deploy the functionality but it's not wired up to control it's just making predictions but it's not actually like acting it's there just uh silently observing and making predictions and then we sort of test it out without it actually driving the car and so in some cases you can also do that um so to me this is just basically the idea that we understood the previous software is is false and fundamentally you just need extremely good evaluation now in those evaluations i'm curious as ever any of the testers or you experienced something they're really surprised by and like wow this car is smarter than i thought um i mean basically every time it drives me around in the latest uh wholesale driving beta builds um and just the emergent sometimes you know just the emergent properties of how it handles uh different uh situations like there's a bicyclist an oncoming vehicle and if you program it properly and then your work works very well you'll get these emerging behaviors where it does the right thing and uh so i would say like every drive i have maybe a few of those um wow yeah yeah that must be a real thrill i gotta imagine you still hold your hand to the steering wheel on your foot on the brake pedal just just in case oh absolutely yeah so the current system is the full self-driving beta build um that i drive around every day and it's actually quite capable i think people sort of understand that the autopilot is um you know works quite well on the highway and a lot of people use it and it can keep a lane on the highway but the latest builds that we have in the full soil driving package are quite competent even off highway in city streets so i was driven to get a coffee this morning and back to my house a 20-minute drive or in palo alto and it was zero intervention drive and so uh and this is a relatively routine for us so it's not a perfect system but it's really getting there and i definitely keep my hands on the wheel because there's still yeah you know we will still do um not very clever things once in a while and so there's definitely more work to be done now of course whenever it makes a mistake in some sense that's that's high value assuming the person takes over correctly of course because that gives you the most valuable data the missing pieces of the puzzle yeah that's right so interventions are are very helpful source of data for us um and you know as i mentioned there's a lot of other ways that we can also get get um data that where the network is misbehaving a lot of disagreements for example with the human driver like we think there's a stop sign we should be stopping but the person just went uh we can look at a lot of that data and maybe half of the time it's people just running a stop sign we see a lot of that half of the time it's hey there was a stop sign but actually it was not for you it was worth for the oncoming traffic and the stop sign was just angled way too much and it looked deceiving to the neural network and so both would be coming back in this stream of data now another thing that um i've heard you talk about and that just sounds really intriguing ties into all of this is this thing called operation vacation sounds very intriguing who doesn't want a vacation um what is operation vacation yeah so um in the process of iterating on all these predictions in the team we are noticing that more and more of its components can be automated so as i describe the process you need to um your neural network makes predictions you need to source at scale mispredictions annotate them correctly and put them into training set and retrain the network that's the loop and we're noticing that you can involve engineers less and less in that loop and through a lot of automation now it's not all the engineers that get to eventually go on a vacation once we've automated the whole thing uh because there's a huge amount of um there's a large data labeling team who who has to stick around monitor your triggers and annotate the data but the actual software engineers who write code could in principle go on vacation uh having automated all the pieces of this improvement loop so i would say it's kind of like a half north star for the team where once these neural networks are just getting better by themselves with a lot of people in between but just data labelers mostly um we get to all go on a vacation and the autopilot could in principle just improve automatically are you worried though that that you all might let you be on vacation for the rest of your life we may be able to get away with a few days we'll see i'm not sure it's it's so interesting because it it also reminds me of when when you were actually visiting cover and you said something along the lines of the data annotation is what you spend all your time on and the data annotation playbook is is so valuable is is the thing that that generates so much value right which is something that somebody in academia of course would never even uh pay attention to um but operation vacation sounds exactly like that that the people who are still working are the ones who are working with the data and everybody else is is just on the beach i guess yes i mean it is it is it is done half jokingly actually as i describe the system there's plenty of design and engineering that can still go into the fundamentals like as an example the system right now makes all these intermediate predictions and there's still a lot of human driven human written code on top of it and this human written code is very difficult to write and it's brittle and it will not fundamentally i think scale to where we need it to be if you really want 99.99 of accuracy and comfort and um so i think there's a lot of there's some challenges that sort of remain i would say on the modeling front and so we'll be busy with those but the fundamental if you're just talking about the perception system itself i think its improvement can be just and just improved autonomously just on the road detection itself and but yeah as you mentioned data annotation is not something you would do as a deep learning engineer we spend a lot of time on it i actually have an entire data labeling org that we've grown inside tesla because this is so fundamental to what we do a typical approach would be that you outsource it to third parties we don't do that we have a highly professional highly trained workforce that curates our data sets and we think that this is the right way to go because this is just a new software programming paradigm again these are our new programmers in the software 2.0 land and so they have to when they're annotating examples they're telling the system how to interpret the scene and they are quite literally programming the autopilot so we invest quite a bit into the org and we keep it close and they collaborate with the engineers very closely that's amazing now when when i think about data annotation i can also i mean immediately the other thing i'm thinking about is self-supervised learning which has made a lot of progress in ai in the last two three years both in computer vision and in natural language processing but probably here the vision part is more important and so i'm curious about your thoughts on the role of self-supervised learning maybe you can first define uh for our listeners what is self-supervised learning and then and say a bit about cs yeah so here's the here's the issue with the current technology basically is i can get almost any arbitrary deduction to work at this point and this is just technology but i need tens of thousands of examples for it so if i need to recognize fire hydrants absolutely doable i need 10 000 examples 50 000 examples and i need to do a bit of data engine to pad out to the data set and i know this will work with a with the neural network this is just technology but there's a lot of things you want to recognize and it feels silly to have to redo this work of like hey 50 000 times this is a fire hydrant from all the possible rotations and all the possible brightness conditions it just seems so silly um and so this is you know this is where the analogy again what the human brain breaks in that for a person you show them a fire hydrant and they sort of get it it's the yellow things on the side of the road that's not how our current technology works it it needs a really good coverage of fire hydrants and so that's why a lot of people are perceiving basically this um there's almost like flaw with the technology right now and they're trying to come up with ways that will not require that huge amount of annotation um so maybe with very few examples the the neural network just like a human network should already sort of like know about fire hydrants and you're just telling it hey that yellow thing on the side of the road is is um you know you don't need 50 000 examples you need very very few because the network already sort of understood fire hydrants and now it's just getting a label for the thing it already has a neuron for and so it's much more efficient at the use of that dataset so so andre when you say the network already kind of understood fire hydrants we was never told what what they are but where does that sort of understood it already come from exactly so that's where self-supervised learning is about is how do you train on a large collection of examples that did not have to go through human annotation effort we so maybe people didn't go in and they didn't put bounding boxes around fire hydrants maybe it's just a huge amount of data and fire hundreds are featured in some of it and uh maybe there are other objectives than explicit matching of human annotation that we can use to to pre-train these networks so that um they develop these representations so there's many ways that people are trying to arrange it as one example of many um that seems to work relatively well is for example you could try to predict the future um and so it's not that we don't use labeled data it's that we are using data that is annotated for free because when you have a video of what happened in the world you serve half the future and came for free in the data stream so you're still training the network to predict a thing uh just like normal supervised learning but you happen to have that thing for free without human effort and so self-supervised learning is a class of approaches that try to leverage the structure of the data in and try to take advantage of the free supervision that we have just in raw data streams instead of uh to to sort of uh get the networks to to arrange themselves into configurations that they kind of understand the world um so that it's much more efficient per label to train anything else fire hydrants might not be like the best example but uh yeah as an example to predict the future uh if you want you have to actually understand the layout of the entire scene and how people and cars move and interact and so uh this prediction problem forces the neural network to understand that hey there are cars they move around there are people they move around they avoid these areas and so when i need to predict the future i need to actually parse the scene to do that properly and so yeah there is a class of approaches and we have tried a number of them i do find that in these incredibly valuable applications uh just paying for the labels is often the right way to go instead of paying researchers and uh but i think i basically kind of agree with that um um yeah this is it's not ideal and uh there are some uh techniques that as you mentioned are seeing quite a lot of traction and we have experimented with a number of them internally at tesla that's really exciting because i mean the way i also see is that once you go self-supervised you you can use infinite data effectively because all data works right you have to be careful though because more data is not always better if you if you add boring data into your data set you may actually be diluting your data set right because your neural network will train for like i mentioned three weeks and it's going to see some number of examples in a training in this training and if a lot of the examples are boring it's not going to learn a lot so you actually want to be very careful with the and this is why i talk about curation of data sets it's an active process of adding hard examples and subtracting the easy examples often and a very easy way to see this is of course if i had just a single image in my massive data set of course that's not helpful so you really want to pad out the variability and that's why i use active terms like curation when i talk about data sets it's an active process to to curate the data set one thing that tesla has also announced is building their own chips for ai compute why why does that matter so there's many possible answers to that of course uh i think to a large extent elon sees ai as just a fundamental like pillar of of a lot of this technology and and wants to invest into internal teams that develop a lot of this technology and co-design everything tesla is is definitely about vertical integration and squeezing out a lot of juice from the benefits of that so to a very large extent of course we own the entire manufacturing of the vehicle in the factory and then we own a lot of the pieces of okay the hardware itself how's it pointed all the design decisions and we own the cluster we own the data labeling team and also we own the inference hardware the chip that actually runs the neural network on the car to us is just another another opportunity to co-design everything specifically for the purposes of self-driving and so the chip is designed with with the kinds of neural networks we want to run in mind and the hardware itself is just targeted to the operations that we want to run and do that do that very efficiently and so really it's just a it's a theme of tesla and it allows us to co-design all the components to work together towards a single goal in this case uh full cell driving when i think about chips for ai compute i didn't think there is training and then there is inference as you alluded to which is when it's used for driving um are you using both is it just inference right now yeah good question so as you mentioned you will typically um i guess hardware for deep learning actually kind of has like two broad areas now uh there's hardware you would use to train your networks and that looks very different from the hardware you were you might want to use to run a neural network um so running a neural network is computationally much more straightforward thing you're just kind of like doing the neurons just have to fire in a sequence if you're training a neural network there's a lot more that goes on there because you have to run the neural network forward but then you also have to implement the back propagation algorithm and you have to run the backward pass and you have to update the weights and there's a lot of technical details as to like at what precision do you run all this mathematical precision in terms of the numbers involved and so there's a lot of details that make the training a much more heavy endeavor and the testing the inference uh a much simpler endeavor and so as you mentioned we currently have a chip for inference uh that is that we own and we've designed and is in all the cars and we are also working on a training computer and this is a project dojo and elon has sort of alluded to it on a high level number of times and it is really just pushing this code design even further and we have a rough understanding of what these computational workflows look like for training neural networks for the autopilot it's a massive amount of video training and um we are building a chip directly sort of designed for that kind of a computational workload and so yeah that's absolutely an active project currently at tesla i'm curious about your thoughts i mean you were at open eye you're one of the founding members and open the eye recently then um somewhat recently raised billion dollars with specifically compute in mind and so i'm curious about that strategic angle also is that is that something in your mind that more compute is the only way to succeed in ai yeah i think more compute is one of the fundamental limiting blocks right now for a lot of things for a lot of domains openai is right now focused on for example natural language processing for example with their most recent work on gpt so what they're doing there is it's a language modeling task where the neural network is generating language text and so you can get you can feed the text and it will continue text or you can ask it to produce text with certain properties or it will answer your questions or it will talk to you and for so what's happening there is the algorithms again in this setting are actually quite well known and understood as you mentioned the neural network takes the form of this transformer you're training it in a very kind of standard regime with back propagation stochastic gradients and so on so that's understood so the algorithms are not the bottleneck for them the data set is also not a bottleneck for that uh class of problems because we have the internet with huge amount of text so in that regime so you are not upper bounded by data sets but you are upper bounded by the compute available to you which really restricts the size of the model that you can actually use and like i said in deep learning we are blessed with algorithms that seem to continue to work better and better as you just make them bigger you're literally just adding neurons into the system and it works better and so openi is primarily gated by compute in the setting if they could train a bigger network it would work better and that's not the way it used to be in ai we used to be bottlenecked by algorithms and so what a beautiful place to be if they could just run a bigger network it would work much better and the results would be even more magical and is that true for tesla also is that yes i would say so neural networks have this property in general that um yeah if you make them bigger uh they will almost always work work better and you know you um and you know in the limit you can for example use this is slightly more technical but you could use model ensembles you could use dropout and a lot of techniques to basically make sure that uh that these models work better when you scale them up and so we are also limited by compute uh to a large extent and we have to be very creative in how we squeeze out all the juice from all the flops that we have available on the car um and so that's the case also on the car but also during training for us right so you want to train as big of a network as possible and for us also um you have to consider the data center to whatever extent that is a bottleneck and the algorithms and the models and to whatever extent that is and so for us for example we do do a lot of manual labeling but we are also looking into ways that you can um train on um data without having to label with a human or you can use sensors expensive sensors to annotate your data and um so maybe maybe you have a few cars to drive around with say radars or lidars or any other sensing suite you want that gives you extra information about the scene and that can function as annotation for computer vision um and so computer vision can be matching those sensors and imitating them and so you have sensor annotation human annotation or self annotation like predicting the future and so all of those are knobs and kind of algorithms you could play with tesla is not the only company trying to build self-driving cars there is other efforts out there and sometimes at least in the media it's depicted as a bit of a race of who's going to get there first and so forth and so how do you see the tesla effort different from the other efforts it's a very good question because it is very different and it is not obvious so for example there was a there was a video just recently released where someone used a waymo car and the waymo drove them to some location i forget the details and then they use the tesla autopilot full self-driving beta build and it also drove them there with zero interventions and so both cars took the same route and got to the same spot with zero interventions um and so to a third third party observer just looking at this these are cars they take right turns left turns they navigate you to where you need to be it looks the same but under the hood the systems are actually extremely different quite different so the approach of waymo and many others in the industry and i would say in the industry we'll see these two classes of approaches really and one is way more like and the other is tesla like i guess in my sort of like description of it i suppose and in the waymo like approach you are going to first outfit the car with many more sensors um in particular the use of uh quite expensive lidar sensors that are on top uh they give you range sensing around you and you also have these high definition maps so you need to drive around before you make the trip and you need to pre-map the environment very high definition and then when you are driving you know exactly where you are on that map so you know exactly how to stay on it and how to drive and this is very different from what the tesla car is doing because first of all we do not have very expensive sensing we just have a few cameras that give us surround view and and by the way that's already a lot of information because each camera is say several megapixels and so you're getting many millions of observations of what's around the car when each ray really is of brightness is telling you something about the world so you're getting a huge amount of information from cameras that is very very cheap and economical to produce um and we do not use high definition maps so we have very low definition maps that are kind of like a google map so it's telling you that hey you should take right turn left turn et cetera but we do not know to a centimeter level accuracy where the curve is everything is coming from the system at that time through vision and so the car is encountering these intersections and these areas for the first time basically as it's driving around and it needs to look at the images and decide these are curbs these are lane markings this is how many lanes there are this is where i should be to take a left turn and so it's a much higher bar much harder to design but it's also much cheaper because the sensor suite is just cameras and it's not specific to a location that you had to pre-map so our system is very cheap and it will work anywhere what this allows you to do then is that this affords you scale so waymo can have maybe a few hundred cars or something like that we have millions of cars and as i mentioned scale is incredibly important to getting ai to work because everything is about data set curation and so i do not see how you can fundamentally really get a system to work well in absence of scale and so i think i would much rather give up some sensing in return for scale in ai problems i'm kind of curious when when you made your decision to to go to tesla i mean you must have seen that bifurcation and was that something on your mind at the time that you thought about a lot about what you believe is going to be the way forward absolutely i i definitely saw the bifurcation and um i felt like tesla had the right approach fundamentally and i'm a huge believer in deep neural networks and their power and i think images provide you with a huge amount of information and it's just a question of processing it and these deep neural networks that i know are capable of doing the processing that we need of them and so to me it's actually a brilliant strategic decision from elon and i was absolutely on board with a vision only approach and i do believe that the system can be can be arranged to to process all that information and actually drive around have you ever had to sleep on a bench or a sofa in the tesla headquarters like elon uh so yes uh i have slept at tesla a few times uh even though i live very nearby but there were definitely a few fires where that has happened i found i walked around the office and i was trying to find a nice place to to find uh and i found a little exercise studio so there were a few yoga mats and i figured yoga mat is a great place so i just uh crashed there and it was great and uh i actually slept really well and could get right back into it in the morning so it was actually a pretty pleasant experience oh wow i haven't done nothing in the while i had not expected this to be the answer cool wow so it's not not only elon who sleeps at tesla every now and then yeah i think it's good for the soul you want to be invested into the problem and you're just too caught up in it and you don't you don't want to travel and so on you just uh i like really i like being overtaken by problems sometimes when you're just so into it and you really want it to work and sleep is in the way and you just need to get it over with so that you can get back into it so it doesn't happen too often but when it does i actually do enjoy it i love the energy of the problem solving and uh i think it's good for the soul yeah so i'm curious what what's your view on the future of ai when we think beyond self-driving what are the big things on the horizon for us i think like first of all like wow the progress is incredibly fast when you're zoomed in to the day-to-day and the different papers that are coming out on the scale of a week maybe sometimes it can feel slightly slow but when you zoom out like alex net as i mentioned this uh this image net recognition benchmark that was beaten by neural net that really started the deep learning revolution and transformation was 2012. we're in 2021 so it hasn't even been a decade and i'll get to live hopefully four more decades or something like that maybe so if if like from 2012 to now has been a complete transformation of ai and a lot happened in a decade and so if i'm going to witness something on those orders of magnitude in next four years it's really mind-boggling to extrapolate and fundamentally we have these algorithms that seem to be on upper bounded by the data and the compute um and we're going to get more compute and we are specializing all of our hardware to neural networks and all that is ongoing our current processors actually are not very specialized for running neural nets and there's a lot of long-term fruit there um and so and also the size of the field has grown and so there's a lot more brain power going into improving everything and so there's this exponential like return on all of this investment in hardware and software and so you shouldn't expect linear improvements you should actually expect like some kind of an exponential improvements so it gets even more mind-boggling and so i think in the short term we're absolutely going to see much more automation be it self-driving cars or drones or warehouses and uh and so on um that's very easy to predict and um but i think on the long term that's where it starts to get kind of even more um dicey because uh you know like i joined open ai openai is basically a agi project artificial general intelligence so the idea is we're trying to develop fundamentally a artificial brain that thinks and wants and acts and functions like a human being so i would say next to a visual cortex we sort of have a check mark like that part of the brain sort of maybe like understand the principles of but we certainly haven't understood the entire brain and how um you know decision making is done and so on but i think we are with robotics and so on we are we are probably going to make a massive dent into that over the next decade or two or three and uh yeah i i think we're probably going to see some very exciting things come from from ai because the technology is not really upper bounded in any like real way um and it's mildly concerning but kind of exciting so i think we'll see what happens andre it's been absolutely wonderful having you on learn so much thank you now if anyone listening is like me and would like to keep learning from andre i highly recommend viewing and reading all the material on andre's webpage carpathi.ai this includes his talk at tesla autonomy day where he's on stage together with elon musk and i highly recommend following andre on twitter where he very generously shares his latest insights on ai with the world and on twitter that's at karpathy [Music] [Music] you

Channel: The Robot Brains Podcast

Views: 20,077

Keywords: The Robot Brains Podcast, Podcast, AI, Robots, Robotics, Artificial Intelligence, andrej karpathy, karpathy, Tesla

Id: PlLmdTcsAWU

Length: 80min 19sec (4819 seconds)

Published: Tue Sep 21 2021