S3 E7 Nando de Freitas of DeepMind joins Host Pieter Abbeel: Generalizable AI to Benefit Everyone

[Music] Our Guest today is Nando de Freitas Nando is one of the world's leading artificial intelligence researchers born in Zimbabwe bachelors and masters from the University of the whitwater surround in Johannesburg PhD from Cambridge Post talk at Berkeley professor at the University of British Columbia Vancouver professor at Oxford founder of dark blue Labs which was acquired by Google deepmind where he's a research director today Nando has won a number of best paper awards at the top AI conferences some of his recent Works include gauto the first multimodal multitask multi-embodiment generalist agent competition level code generation with Alpha code learning to communicate learning to learn and playing hard exploration games by watching YouTube Nando is very active as an educator and Community builder for deep learning and artificial intelligence in Africa and South America Nando so great to have you here with us welcome to the show it's so exciting to be here I'm a huge huge fan of this podcast and so yeah I feel privileged to have an opportunity to begin thank you I'm so honored to have you on now now before diving into today's conversation I'd like to thank our podcast sponsors index Ventures and weights and biases index Ventures is a venture capital firm that invests in exceptional entrepreneurs across all stages from seed to IPO with offices in SF New York City and London The Firm backs Founders across a variety of verticals including AI SAS fintech security gaming and consumer on a personal note index is an investor in covariant and I couldn't recommend any higher working with them weights and biases is an ml Ops platform that helps you train better models faster with experiment tracking model and data set versioning and Model Management they're used by openai Nvidia and almost every lab releasing a large model in fact many if not all of my students at Berkeley and colleagues at covariant are big users of weights and biases now Nando AI has become mainstream it's something the world seems to have especially with recent advances in chat GPT and so forth accepted as something that's really part of what the world is didn't used to be that way it's a big change from our early days and in the machine learning community what do you think this will mean for most of the population on this planet it's hard to imagine it's I mean first of all it's it's been such a big change since um and we used to go to Europe's together and Solomon and indeed now everyone on the street is talking about AI so it it has become part of I guess the universal conscious and um what will AI do for all of us I you know I like still my favorite way to think about it as I think of it as a tool and um and alcohol or a family of tools that will allow us to do more just like microscopes and telescopes extended the range or personal computers for that matter extended the range of things we can do um My Hope Is that AI will extend the things that we can do whether it's to create new knowledge whether it's to solve some of the hardest problems that we Face environmental problems energy problems um and above all um I would love to see it um Being for the Benefit of all humans not just one particular group but really for humankind I think it's a natural progression that will be in ultimately very important I think in sort of the very long range of um many centuries of Millennium to the future will be essential for our survival so I think it's it could be a wonderful tool and provided that we we use it wisely and safely and with compassion now you are at one of the main people said the leading Institution for AI research today Deep Mind Right based in London at the same time you talk about something that you hope will benefit the whole world um yeah how you're going to make sure it plays out that way um so one of the things that I've done um you know I guess each of us can do a little thing and I think it's important um you know admittedly I would not know how to do it but all I could do is do a little bit and um and I've personally been engaged in volunteering so I love teaching so I've continued doing that because that's the one thing I can do and so I helped with the creation of the indaba you know raising funding Yourself by the first two in doubles I even helped going over the coding exercises and simplifying them and so on for the labs to run and have runs the labs of the first novice and also um by the way it's it's one of these meetings that we offer in Africa it's a gathering of people students startups and it's aimed at promoting artificial intelligence and the use of technology in Africa increasing access for the African continent to Ai and that also inspired the creation of several other efforts across the world one that I'm also involved with is kippo in Latin America so I also attended the first keeper and in a few weeks I will be going to my first meeting in in three weeks it's a sorry three years it's been a long time since I've traveled the conference um and I'm really looking forward to it because it's being with the students in Latin America or in Africa it's extremely energizing and it is so important to go there beyond the content Beyond talking about Covenant so um uh whatever the the flavor of the day is in AI um when students see you and sometimes we forget um who we are in AI like you know Peter bill you know one of the greatest Stars I've always admired your work and so on and millions of people admire you as well so if you come to Latino America if you come to people in person and not just give up um hopefully when they're in the future and this is true of anyone in AI I think most of the people you've interviewed in this show just by being there and the students saying that they can have a conversation with you and they actually realize that hey this person is um Lord as clever as I thought I I you know I was just as clever as this person and I can discuss research with them and that gives people a sense of empowerment that allows them to realize that um I I can also achieve that I can go far and it and I know this because I was one of those kids in Africa and I remember talking to um Professor Longview for instance control guy from a Imperial College and for me meeting um as a student a professor from Imperial College was it was like I don't know meeting the Dalai Lama or um you know it it was an incredible transformative moment because that's when I realized I could be part of that dialogue and I couldn't over emphasize that but it's the first thing you can give people it's just the ability to believe that they can do it that they can be part of it and so that's a little bit that I do and I encourage everyone watching this podcast to consider attending um people or um in Dhaba or some of the meetings in Southeast Asia and throughout the world it's really important um I guess for us as especially senior researchers in AI to do that um and because that just allows for the community to grow uh there and it's also important to kind of talk to the people there and the startups there's a whole ecosystem that you want to grow and and it's also important that then that ecosystem starts growing and that it's the people there that start leading it and directing it and I've I've Loved how they end up I grew and it led to many localized meetings the end of X-Men things all throughout Africa and it's It's the Most Wonderful Time of the Year when you go to one of these meetings and you you're able to engage with so many young ambitious uh people it's interesting to bring it up because a couple of things come to my mind now on the one well you you're clearly snuck an invitation in there for me I'd love to take you off on that someday even though this year I probably want one travel um yet myself but I look forward to it um other thing that comes to my mind is when you talk about the energy the excitement while I haven't been to one of the events in Africa or South America yet myself um the black and AI workshops bring a lot of the researchers to nureps right and that Workshop has tremendous energy and I have to imagine it's a bit similar to what you're talking about and it's absolutely amazing the energy that's there very much so it's wonderful it's one of it's you know we talk about the Transformations that we've had by in the field by going into you know brought in by the gpus or brought in by the Transformers but I think the people transformation has been equally amazing I really know where our community is heading um I think there's a lot more work on that we all have to do but the the work where we're moving in terms of diversity and inclusion I think it's it's been one of the most important things such as happened to the field and I just hope we can all continue to move in in that direction towards making sure that um our tools our research and so on capitalizes on what everyone can offer on diversity and and to ensure that it's it's an inclusive technology that we're building towards an inclusive community um I yeah I think that's essential for what we want to achieve in the long run now if I can riff off of that um if I look at the current trends in AI one of the trends is building very large models right and building those models is very expensive so even though you know we might educate people and so forth educate and might not be enough need to actually give people quite large resources in one way or another to be able to do those kinds of things um or or maybe there's other ways I mean obviously the large models are not the only thing happening but it is where a lot of the excitement is so I'm curious how you think about that how that that Trend interacts with everything you just said yeah as I think you know we answered the question um the large models is not everything I think the large models is a very exciting um direction of research uh I mean we all know that extra compute allows us to build um bigger models and bigger models to uh tend to have properties that smaller models that allow us to do certain kind of research that we can't do with small models so the large-scale empirical work that's going on in the world is sort of very important is in his advancing Us in big leaps um but at the same time um there's a lot of innovation that can happen outside and that Innovation can be at in fundamental work and we see this happening all the time I mean we all talk about the tension as attention is um this corporate thing but I do remember there was a student that came up with this thing I remember interviewing the student in Oxford that is a attention idea effect and and then then went on to I don't know and then it went on to Montreal and then developed the idea further with researchers in Montreal and eventually you know it took many more iterations until the technique was fully uh you know it was gave birth to you know the Transformers that we use now and the gpt3 but it starts with the student coming up with this idea and and there's a lot more of these types of ideas you know you know recently we see these structures database models and um and and we also see a lot of work with differential equations with it um there's a lot of things in terms of fundamental work that I think people um can do there is other type of word that I think is very important which is to um how do we use the tools because even if we do have the good language models and it eventually I think we were going to learn how to serve them more efficiently and allow people to fine-tune them one way or the other um and if people can fine-tune them efficiently and and have access to enough compute through the various clouds that you know different organizations offer um then I think it it does become possible to then address the next big challenge which is what we do with these tools how do we make these tools useful in our communities because when you go you know I go off and every year I go to Argentina so on I have family there so then I go to South Africa as well and when you're in when you transport to those countries you encounter a different reality which is very different than say what you encounter in Berkeley or when I encountered here in London so Nando I like how you're saying that you know different parts of the world have different needs challenges and hence different opportunities to build AI applications AI companies for that matter can you give some examples I can and this is probably another reason why it's useful to travel and volunteer and and attend meetings all over the world um when I went to the festival to give you an example I learned about this phone app using like cheap phones that most people actually have access to and one of these text apps is used to provide Better Health Care and one particular example um that I thought was very poignant was um where often moms will try to text a doctor to to ask a question like for example why is my my child hasn't been able to eat um I give my child water my my child uh you know vomits so it's not able to stay hydrated it's been one day or and so on what do I do and sometimes all it takes is a you know a simple message and some pieces very simple piece of advice just like boiled water and and to be able to help that person and because a lot of people in around the world are dying of dehydration you know very simple diseases that we don't even think about in the UK or the USA um and so yeah just sort of and so then the the next day out question for the startup was how because we don't have enough doctors how how do we try to automate the system so to be able to and and this is actually something now I I would think with language models we do a bit better which is to provide help to people who are in need of that help and you know it's about enhancing every what people can do so one of the things I would hope of AI is that AI enhances from every person to become to some extent better capable at providing first aid help or being able to deal with some medical conditions and and and I think yeah that was a great example or a problem that I don't think about because I never encountered it here but then when you go there you realize it's a big problem and then you realize that with some Innovation you could actually save many lives and of course then there's many other problems that we don't even think about here you know there's different economic systems there's different security challenges there's different um um economic problems things to do with Agriculture and so on and and there's and it's only by big part of that world by going there by talking to people that one actually learns about all these very interesting research problems and is able to actually um contribute and and yeah and hopefully be able to um also really seeing the true benefits of our technology uh you know helping most of the people on the planet I really like how much time you spend them um just making sure you reach much more of the world than than most of us end up spending time on and and then making it clear how we can also contribute Nando um I feel a little pedestrian asking the next question and encounters to everything you just talked about but and I guess for people like us a lot of the work that we see happen is coding and coding can be very time consuming and frustrating at times because most people have a clear idea of what they want but then somehow computer won't do it exactly right um and you recently um were one of the big contributors to Alpha code at deepmind can you see a bit more about Alpha code first of all it was a wonderful experience um to be part of its first uh the project done by a large team and um it was wonderful to witness um what what people were capable of doing um and so Alpha code um is so every year um many schools burka is very good at this and um um compete internationally or nationally and so on in these coding competitions where you you're given a problem in English um um you know for example he I don't know you have um certain goods and you want to distribute them one way which is the best way to Optimal way to solve that problem and and then a very smart you know the smartest students at coding yourself from all universities get together in teams and then they go to these competitions and then they quickly try to hack some code that solves the problem and and so strong a perspective of doing research in AI it seemed like a very interesting problem of uh to it's a good challenge um because it's very easy to measure and so on um and and it's a wide interest how could we go automatically from that description to code that solves the problem and uh and so Alpha code was an attempt of doing this it capitalizes on using um no surprise here Transformers and the take that description is input and then generate run the code it uses a few other tricks like sampling many solutions and uses a lot of very clever ideas as to how to narrow down the samples to so that we can uh um solutions that are able to you know so we so we are actually able to compete and and get a a reasonable success rate of writing code that addresses those problems and it's been truly it was truly remarkable how well it did it wasn't perfect um I myself was looking at a piece of code and sometimes there's a I'm reading the code and then there's a for Loop doing a bunch of things and then there's more code and then you realize why is the score look here and you realize it's just it well it's just extra code that the machine created and that is not just anywhere it's a bit like I think grad students to throw everything into exam and some of it will be the solution in one operators um things that they were thinking about well um and um it's not perfect but I love those um there was a Blog that actually says that it's like it felt like there was a dog in the room and um and the dog was speaking English but everyone was pointing the finger at this dog for not having proper grammar and then um yeah that blog I think creates were best described uh how I felt about it it was just for me it's just it was amazing that it could do that because I was a bit involved uh with the student so in these coding competitions at UVC 20 years ago so to see a machine doing that um is it's just quite remarkable and I think even one day perhaps these machines will move on from having a performance of say being in the top 20 and software actually being some of the top colors and out of that there's also besides just sort of that challenge and that also has led to a lot of people and not just the mind but also uh many other places including of an AI and so on to come up with tools to improve the the workflow of uh software engineers and coders um and I oh I find that remind I mean I find myself nowadays like I type A P and it completely writes the whole print statement that I had wrote about when I was debugging something and it's it's here it feels like oh it's reading my mind but um it turned out to be a very useful tool yes becoming very real I talk with my students and they have this code completion uh add-ons turned on pretty much all the time it doesn't mean it always auto-completes correctly but very often it doesn't and if it doesn't then you know they can always not accept it and apparently it's speeds up their work tremendously as they're uh coding for their research projects it does especially if you're not coding every day and you just dive in every week maybe a couple of hours then you always look at Rusty of the syntax but this thing just solves that problem for you it's it's quite amazing how how much it can improve your productivity now I think one of the things that's really interesting about the agents that can help code is the generality right it's it's something coding is such a general thing um compare that to something more specific like learning to play maybe a game of chess or learning to play the game of Go those those breakthroughs were very interesting because the methods were shared across the different games in the end but the agents were not shared it was a separate agent separately trained yes with the same method but separately trained building its own neural net essentially to get the job done in a separate agent separate neural net for the next game and so forth and coding is much more Channel and in fact a bunch of your work is really pushing the frontier of generality going from these specialist agents that are hyper trained on this one thing and sure maybe can compete with humans on this one thing really well but then it doesn't mean they really understand anything else and so I think they gotta work that you recently published is a great example of pushing the generality to a whole other level can you say a bit more about what you did in that work and also May what inspired you what made you think that this would be possible because to me it was surprising how well it worked and how it was such a general agent doing so many things with one agent yeah I mean it's I think for many researchers that's always been one of the dreams is to be able to um design one one one single intelligent system that is capable of doing um and I think what captures imagination is that can do anything that a human can do for example and perhaps you can do other things and maybe it doesn't slightly different or completely different than how a human would do it but we've always aimed for this generality in fact you know people started calling AI AGI because to emphasize the importance of um generality um and I mean as you know I I was one of the people that attended the CFR meetings where we found the sort of deep learning came from and right from the beginning and and I think within that Community the uh within the Canadian Institute for advanced research where you know I think you were attended a few of those meetings and Andrew Aang and you know all these wonderful people he had in your in your meetings um um there was always a connection with neuroscience and so the moment you think about Neuroscience you actually realize well this piece of tissue is sort of very general it was a piece of brain tissue it can be used for perception or it can be used for action and and even you see these implants where you can implant something with pimps in your mouth and with a camera um if you're if you you know cut your fish and it's with time through this pin and depth depth processing you actually start being able to tell depth uh without seeing and and funny enough is the visual cortex that gets recruited to do this still even though the sensation is tactile so this is haptic it's no longer Visual and so and and of course there's many Works in Neuroscience it's a very old idea going back to the 70s kind of pinpoint this that um that's perhaps just one universal algorithm that can do it all and and so then it does beg the question when we have these new architectures and advances um are we now at the point where we can revisit that question and um how we would we go about doing it and I was fortunate enough to how um encounter and in my time at deepmind are colleagues called create who is very passionate about the same things and here is um incredible at executing plans uh and executing ideas and it's also fortunate to have this brilliant team of young people who are very open to make this happen and and so we spent three years on working on this and um and at many points we sort of forward you know going for Publications and so which is often that drive the vehicle such as how and and just to focus on sort of bringing in more and more doing some of the boring stuff like let's bring in more and more data sets let's build more and more infrastructure and um that sort of those up and downs in that process because sometimes you wonder are we should we not just like speed us up and just of that and um but so that's how the project came to be and we just aim to see whether we could use um come up with a single model um single big neural network that would be able to take all types of input uh whether it's patient whether it's dialogue whether it's Peach whether it's you know torques um you know Pro perception and so on and be able to either talk to you or control a robot arm and so and eventually well you've seen the results we've had some success with that um I think but I do see that as a first step um I also don't see that as the only work um very grateful for your thinking of it that way I think it was one of the worst but enjoy yourself came up with a decision Transformers which is a very uh a similar idea in spirit and and I think there were other ideas at the time so Galloping and so on um but as I said I think that's been a drive in machine learning for for many years and in fact I remember discussing this at several with Andrew ing who was also very interested in these types of mobiles and in fact that was at the time when just before he went and started Google brain and it was believing in the Stream the kind of draw that I think to some extent and I think we still have a long way to go but that that dream is still alive and and you know I'm I'm happy that people must see that it it may be possible and that there's more and more people interested um in working on it and and of course there remain many problems the remain many challenges many things we haven't shown and um but yeah it's definitely something I'm excited about and that I've learned to continue working on and I think many others will purchase um including yourself are working on it now and by the way your recent uh paper I have to um you know put an ad here as well for all researchers to kind of read features of your latest paper generating videos using YouTube videos I think it's brilliant and if it's this kind of idea that I think will be very important to build more General agents yeah it's interesting it's all very related I think in the gato paper that you wrote yes it ties to decision Transformer quite closely but I I would say there's something fundamentally different which is the Insight that you can tokenize everything and that people used to think of multimodal learning as there is this mode and that mode and it's all its own thing and you realize that if you properly of course the tokenization needs to be done right but if you probably turn it all into a sequence of tokens a single architecture can just process it all and and generalize which is which is really beautiful I think yeah it's it's also based on our idea I I first tried to do something like that using a method called paq and so I think a lot of people might not know paq but it was a method invented both text compression and it was for the very long time state of Bayard um and so ppiq would take images and text and all sorts of things that would transform it to sequences and basically sequences or bits until you know if you wanted to be efficient you kind of go to bits and you try to make the code and then it is essentially a context um it was sort of hand engineered features using the context of the string of bits to get a neural network and predict the next bit and then it's an architecture consists of many neurons and every neuron is just trying to predict the next bit it's a very interesting architecture so I became quite fascinated by it at the same time that India was working on his uh let's see on three rnns as a student in Toronto and so I remember doing working in this and and trying to convince earlier that they should switch from his RN hands to be a queue but I'm glad he didn't because that actually was quite brutal but it was but the idea was already there where we were using a single model to deal with images to deal with and an effect at the time I think I was using an RPM or or sparse coding mobile and then and then I was taking a sequence of the the hidden units and then I was seriously doing a PhD with uh um by the way I shouldn't say iOS um a student of mine at the time borrowed no who is now at Global and has continued working on this with similar ideas and he was a brilliant coder and he was able to affect all the stuff in at the bit level bit operations um but yeah so that idea was so over the idea of using it everything should be just a single sequence was already um there in the literature and so um yes caught I remember words casters with them he was very excited about it and he definitely pushed for it and uh he was a champion of this idea Deep Mind and eventually succeeded with it and I think um actually certainly surgeon and his team they also came up with a very similar model where I think they were also tokenizing everything too just use a sequest Transformer so it's very closely related to that I think what we did differently perhaps is we actually devoted a lot of time to it I think we might have started returning some working on it and we just made sure that we brought in as many data sets and as mainly tasks as possible and and that's a big challenge because once you have if you want an agent to do 600 things in environments that means you need to be running 600 environments during your training and that's a massive Endeavor in evaluation and data processing and so on so um yeah one of the the people that led this um to serve their uh and Commerce in in the Deep my team I think they did wonderful engineering work um Gabe as well yeah there were lots of Engineers really working very hard to to make that infrastructure possible to be able to run those experiments talk about generalist agents gato we talked about is essentially um learning ahead of time from its own experiences and then can do things later we also did some work where you don't have to do all the learning ahead of time you get to go watch videos retrieve videos which to me seems brilliant that's that's interesting matches what most humans do if I have to fix something in the house I will find a YouTube video that shows me how am I supposed to you know fix this drain or something I can't do very complicated things but the very simple things that go watch a video and all of a sudden I can do it even though I've never done it before we've done this with AI agents um can you say a bit more about what what kind of capabilities did agents acquire and so that and that only work where we um so it you know it actually started by me watching my nephew playing Minecraft and and and it became very good at it just by watching and I was like that's how we should be learning games we shouldn't be doing reinforcement setting in an environment there's so much content on the web but we should just learn from it and now it's many people doing that and it's kind of people have done it very well with Minecraft recently um but at the time I thought you know let's try to do that let's try to capitalize on everything that is out out there and of course there is a question there no uh domain you know how different are all the versions of Atari in YouTube from the simulator that you have but if you do happen to have a simulator and you have many videos then you can take advantage you can do some Transformations it'll be a small domain Gap but you can deal with that and then be able to sort of try things in the simulated environment and learn to play games so at least for a tally we were able to go and just watch videos of what's going on and then be able to play those games and be able to essentially you know play them much but I think that if we were using um much more expensive RL Agents from scratch um that paper that we did back then that was sort of a taste of what could be possible um I I often end research I like to um find problems that you know that people haven't really tried them really hard and and to just to show that something is possible and because I think that's you know just like when you inspire students and you go there and you just they once they believe it can be done then they they will go ahead and and do it I think with research is a bit like that if we can show that it's possible to do something um then soon after there will be a chain of other papers and products and so on that will far improve upon that um and I still believe that we have a long way to go in terms of taking advantage of that idea and they still all of that video as you said or how to do anything on the uh on the internet and we could take uh advantage of that video um not just to play games but I think ultimately as we're able to um especially now that we're building much better video models of the world um you know with the future techniques of masking Transformers and so on um I think we we will soon um see different realizations of that idea where people really can sort of learn from how to videos in YouTube and they're able to transfer it uh to column um I think that the most obvious thing is how to get robots to do different things because as you know because you have more space in this than anyone I think uh collecting robot data is very hard and we often see these videos yes of people telling operating their hands of the robots and so on but then when you actually go and try these daily operations and stuff it doesn't really work it's very hard to do good tele operation and of course you're limited by how many humans um have the stamina to actually tell the operate robots and and then there's this other thing that happens in the lab which is you teleporting to collect data and then you know your group of breaks down audit or you have to upgrade it because the company has a new crypto and um and any of those changes in Dynamics of the machine or in the appearance and so on just mean that you have to collect data again from scratch and so it's very time consuming to collect data it's also very wasteful to just have to collect again and throw data away and and there is I think even if we were to pull all the data from all the labs in the world Perhaps it is still So eventually you do have to go to YouTube see how people do things and and and then be able to come up with innovative ways of transferring that knowledge so that the robots can actually just watch just like we humans see others doing something and then we do it um I think that ability is that is something that I expect we're going to see a lot more of within the next years um and um it's I I think it could be transformational for robotics I'd be curious to uh but I'm going to turn the interview on it how transformational do you think it will be in robotics if the robots can just watch it even doing something and then are able to go and do it uh what prevents us from um in um based on your experience from doing that is if the question is it algorithmic is it Hardware It's a combination but I I mean I think if we can actually have a neural network that is sufficiently capable to watch a person do something and then know how a robot should do it I think it it really opened up many many opportunities and it could be watching a person live in the same context which would be a little easier or it could be retrieving videos online from related contexts like we as humans do we usually don't have a video where somebody did the same little fixing thing in our own house it was a video in another house and doing some something very similar right um so yeah I think a big when I think about robotics myself it's essentially repeated motion robots solve problem for a very long time building cars building electronics and so forth but that's only two three million robots in the world then you run out of opportunities for these robots because the rest of the world is not so repetitive and then think what I'm doing now warehousing and related things in terms of difficulty farming recycling I think that's the current generation of things that are possible but yeah absolutely I think what you're talking about is the next thing after that and then you you can think once you can do that learn from one video how to do something as a robot I think you can go into houses and be super super helpful there's a question as you alluded to Nando um everything's built for humans are the robot grippers Channel enough to do all the things we want them to do in a house today probably not um but maybe it'll give a very strong incentive for some really great mechanical electrical mechanical engineers to dive in and build some robust human size hands that then could do the similar things so yeah I'm it's hard it's kind of a weird question in many ways for me because it feels like it's been the goal of my research for like 10 10 20 years and it's kind of weird to think that maybe we could actually do this in the next who knows handful of years maybe even sooner um it's uh yeah I don't know it's uh that in some ways the whole thing that is easier than solving robotics because you also have to deal with the Hogwarts question and the hardware question is and there's economic factors that also factor into building machines is not actual machines it's not cheap um I do think it's important it's in um I still think robotics is one of the most important approaches of research in in AI in intelligent technology because I think you know for dangerous activity you know activity and so you know where when an activity would be too dangerous for a human you know where you need robots and these arise especially when when problems arise that humans have to deal with um earthquakes and so on as we certainly witnessing recently um it would be really useful to help machines that could help in those scenarios I also think ultimately um you know for space exploration um it also becomes really important um to to rely on on machines because it's it's very it would be very harsh for humans say to go to Mars and so on but as you know we can already send robots there it's also I think that's the rovers are the first step um but I think if he if you do really One Day end up being an interplanetary species and then uh robots will play an essential key role in that they'll be the enabling technology that will allow us to achieve that yeah absolutely absolutely agreed um switching from robust to a slightly different topic um as if alluded to in this conversation in the past it's particularly interesting too in research to push things that haven't really been shown much sign of life before and all of a sudden you realize maybe there can be some sign of life here let's let's show something new and a work that really stuck with me and influenced a lot of my work is you're learning to learn work you had this paper cleverly titled Learning To Learn by grading dissent by gradient descent and as I understand it I mean the big idea is that why should we write the learning program still can't the learning program also be learned right it's a recursive process in some ways um this paper is now from a couple years ago I'm really curious maybe I can quickly recap what you did in the paper but also give your perspective on what you think about learning to learn today in terms of like my experience with it it started um so I was pointed to this paper by one of my collaborators Mr denial [Applause] operator I think and um and it looked like a wonderful idea when a disability as it was worth revisiting and doing it with our modern uh neural networks I became fascinated by it because I think in as much as possible I I always believe in capitalizing on the data that we have um but it sometimes I also find it interesting just scientific curiosity as to how do things arise how do things emerge um and so and and in particular I was interested in um you know I was philosophizing a bit I was thinking you know thinking of evolution as a learning process that then creates these biological machines that can learn so it's a learning process the dynamical Adaptive process that has led to um the generation of dynamically adaptive processes that are capable of learning amazing things like calculus and algebra so and um and so to me it made sense that this just we needed to you know endow our neural networks with that capability and so we start working on it we tried quite a few things and there was a bit frustrating in the beginning because we weren't getting very far with that um and then um Martin who was very young collaborative mind at a time Martin andrachovich who actually went to work with you afterwards and did a wonderful work with you um he sort of persevered with the idea and um eventually uh got to work um and it was you know to help a few others Sergio and son and uh and and then I became really excited about it uh because I I thought you know there's an opportunity here for uh neural networks to learn what the other neural address should be like what that what the algorithm should be it's like we're still hand engineering a learning algorithm um unfortunately it's still Adam up to many many years that most people use so we haven't been able to engineer anything much better than that down um but um there is the possibility that we could engineer better algorithm so we couldn't have a neural network generate new neural networks that would solve and the problem um and you know that lets some lots of brilliant ideas on learning to learn that there's an explosion of words in that area and a lot of those words actually came from Berkeley um from your collaborators itself and um and more recently um I'd actually just see that manifesting itself with the big Transformers and the language models so that's one of the emerging capabilities of these big models that has really amazed me is that I still see them as especially in the first shot it's like if when you pre-train your model I feel like you've learned that initial thing and then when you give say some context especially if you're prompting it with a few short prompting giving it up to you sort of input example solution input example solution and then a different example you give it a solution I see creating that context whether Transformer and during prompting as essentially basically um you know with your model what you learned was the ability for you to be able to prompt the model so that it's it's learning from few examples so I see that is like a few short learning to learn um at amazing skill and generalities I I think now I see that that idea is actually happening a lot and that's been one of the sort of really remarkable things of the big language models is that is to see this emergent capability but if you start bisecting and doing the math carefully um it's not too far from the things that we were doing before with before you know one shot imitation learning and Robotics uh we didn't quite have the the big models for example I think the ideas were quite slow um of course there was a lot more constraint than what it is now it's just like one single big model doing it in language a lot of the current work focuses on the very large models right and of course they've had some amazing successes and unprecedented capabilities I mean there's a good reason there is so much focus on them right now but there's another line of work which you've actually done some of the early work which still fascinates me which is where the AI systems learn to communicate in the context of having to solve problems together and and hence invent the notion of communication because it helps them be more effective maybe a bit the same way humans and animals have learned to communicate because it helps them survive better helps them get more interesting things done and it hasn't been as active recently but I'm I'm curious I know I'm putting a bit on the spot here because you probably haven't prepared for this question but what are your thoughts Nando on combining some of these things with the current large language ones or maybe even these things possibly in a few years supplanting today's large language models because somehow it's fundamentally more similar to how humans came to language yeah it's definitely worth um revisiting um that idea in this context I think that that would be a good project um for me you know it comes back to this question of do you use all your data or do you look try to understand take a more scientific View and try to understand how did language emerged and uh which is still I think well maybe it's still an opening question we don't know how it is that we got to this point um so if you're pragmatic um you have to realize that we have all this human language produced by the whole human race Through The Years on the internet recorded mostly um so you could take advantage of that to train big models and why should we bother to go into this other type of research it's probably the same reason of why we do history why we do many other things because it's it's an unknown it's something for which we don't know the answer and and I think it's I think many of us would love to know how language comes to be um and so that initial project that was led by my student Jacob burster the professor now in Oxford and niani Society who is also now a deep mind um try to use multi-agent reinforcement learning and sort of created environments where the only way the agents could possibly solve the problems was if they were to send information to each other that was meaningful and they also put some of these bottlenecks like sampling before the soft Max and something to force um the models to use discrete communication if then if the channels were noisy because we wanted to know whether they would come up with um discrete symbols or the script representations to communicate more efficiently to be able to solve the task uh we also tried to see whether they would be possible for them to learn to sort of eventually combining these symbols and compositional ways in this interest in the same way that we compose with language and that turns out to be very hard and I think there was there's been some progress but there's still we still have a long way to go I mean one example that I love is um like in nature you have monk you know like there's this monkey and um I forget the name of uh at some point I knew the scientific name of this monkey so the East Africa but they essentially have different um they have certain monkeys and and then when a Predators nearby whether it's uh uh an eagle or a snake or some um don't remember I think it's a jaguars there too if one of these animals is coming then you they will produce a different sound that is indicative of what animal is nearby and of course the response of the max will be different if it's a needle or you know if it's an animal or a predator on the ground if it's an eagle they can just jump up tree um but if they're on the canopy on top then you know they become sorry because of the tree because they don't want to you know they could be invulnerable from the uh um so they will do something appropriate to uh Safeguard themselves from the predator um and so it's it's easy to build a narrow system where the agents will learn to do that um the thing that was very hard to figure out for me is the next step when eventually one monkey sort of learns to manipulate that symbol or starts thinking you know I I would like to get all this food but that is just not much that I don't like and I don't want this other monkey to get much food and so maybe I will just tell this monkey that it's a tiger even though it's an eagle and then that way I can get all the food oh boy that that um rather deceitful way to stop it intelligent unfortunately but um and lack of a bad example that it's being able to then sort of reason about those Childs to go to that sort of high level of abstraction it's when the just the sound the discrete symbols sort of transcend the immediate material and and then we can start manipulating them and just like we humans use words or we come up with these figure eights that are horizontal we call them Infinity so when we manipulate them and we're able to create knowledge about things we've never seen the universe and eventually we see those things with some fancy telescopes um it's so much of science has come especially cosmology and so on has come as a result of the basmatical liking this abstract symbols we can mix we can make predictions that are so far beyond anything that we can see or will ever experience and how you could create new symbols create new knowledge that I think is still a big open question even and it's it perhaps is going to become very relevant for language models which is not just to um paraphrase that humans do and so on and or sort of just combine with uh what text that is underwear but it's actually um you know language models and maybe sit down with coffee and start thinking about the mysteries of the universe and start creating new symbols new abstractions that allow them to then do some inferences and be able to make some predictions about the universe and be able to also come up with testable ways to verify those on predictions um so I think invariably we will revisit this question of learning language which of course has a lot to do with learning to communicate it's very interesting because you now talk about agents that could actually run experiments right don't just output text but with maybe press some buttons somewhere on a either on a website or maybe physically being credited in a robot try things out in the real world see what happens and from that make new hypotheses and and learn faster hopefully uh than they could otherwise have a deeper understanding of how how things work um that that's really fascinating I'm curious if we kind of zoom out for a moment Nando when you think about the next let's say five to ten years in artificial intelligence what are the things that you are most excited about well I think a lot of the things we've talked about today um I think I imagine everything you've talked about today um I think the the mainstream discussion now is of course um on big language models and of course the questions uh important questions arise there you know how do you make the language models actually be factual inaccurate so this question of Truth um there's questions about how do you make this technology is actually safe um how do you make them truly useful for humans um how do we and you know and then there's engineering questions that we make sure that we imported the general questions and we make sure that we trade them with a lot less energy uh how do we make them also you know cheaper um easier to access and so on and of course all that comes with security and safety um and and also how do we contextualize them because we were talking about um communication and language so there's different types of communication sometimes there's communication where you where like for example you're interviewing me yeah I know that at this point you don't want to just chit chat and I shouldn't just go and start telling you um you know about what I ate for lunch today and spend the rest hour of the day I don't think the viewers appreciated and bad um and and so the context is to a large extent determines what you talk about or um and of course context comes in many ways it could be where you are um it could be who you're talking to and um I think that's something where we need to make progress on and and of course and I do think we're going to see a lot of progress because there's a lot of driving forces that will make us go in that direction um also I'm interested in just sort of scientific questions I sort of think experiments that we're talking as we talked about and and particularly I'm thinking the experiments that we do in our minds this thought experiments like Einstein used to talk about imagine someone traveling has been replied and so on this is something that I would hope one day the language models could do because a lot of the experiments the experiments with metaphysicists do for example cosmologists and so on and the sort of thing that Stephen Hawking with good at and these are experiments you do with pen and paper and so provide us to have a way of using tools of externalizing knowledge of storing externally retrieving this knowledge um and and provided that then you can sort of group things and create new abstractions to be able to sort of talk about uh problems I would hope that we would see a lot of scientific progress and I think those scientific progress with um I mean can you imagine if the machine starts telling us no guys you had this all wrong with how to build Fusion reactors let me tell you I've spent the last few seconds thinking about it I derived the equivalent of Melrose of humans businesses for the last 200 years and here's my conclusions too how you could build this nuclear reactor and you can I've created a simulation when you go and test the verified that it's uh you know check my idea and this is what you need to do to verify it and so on here's some tests um I mean it would be wonderful uh and we're starting to see that a bit in biology there's been a revolution in biology uh right like with um yeah obviously I'm extremely proud of alcohol um and um without the fold we're seeing a lot of followers just getting very interested in neural networks and machine learning and so we see a lot of work on these Nano machines and proteins and so on and um and we're not quite there but I would hope the neural networks eventually with the ability um to think about problems and and reason about many complex variables and be able to um you know derived new knowledge just like we do by reasoning by abstracting um and maybe by advising on what experiment should be conducted What machines could be created um yeah that's what I would hope this will take us because I mean if it takes us to solving our energy problems in getting safe Vision reactive to bring energy to the entire world that would solve you know the problems of time and so on and economic problems um or even you know reason about we're starting to see a lot of uh climate models better climate models and we had in the past and it would be interesting if the models could recently and and help us you know make better predictions as to how we could go about what kind of interventions we could do to ensure that well we protect our environment and even economic systems I think economic and political systems are very very complex even for well-intentioned uh policy makers to to be able to comprehend to make rather than policy so if if the machines could assist people to come up with better economic policies but environmental policies and so on I think that would be wonderful well I think it was sort of the mainstream um at least that's my dream over I would love to see this going and of course a lot of this is that's beautiful now augmenting scientists but I also would like to augment the people who are doing labor so I you know my parents they just went to work and in fact as a kid I would go to work with my dad I worked in construction in this electrician and and it's hard work and for most people on this planet you don't got to you know we are so lucky I go to work to explore ideas and I have this you know I I I'm so fortunate I work in this wonderful environment that is full of these incredibly smart people who love to explain things to me and brainstorm with me and so on and we're just having these big dreams and I get the opportunity to talk to people like you and Christian campaign podcasts and so on but for a lot of people as blind it's able to work to bring home the bacon to um to make money look after your families and so on and sometimes there's a lot of hard work and sometimes you're very tired and sometimes you break a toe or something but you still know you have to go to work and even if you're in pain um and and that's the reality for most people um and so I think for a lot of those tedious tasks um it would be important um to help people with again but that will involve hard work that's where robotics comes in and not necessarily um replacement of labor but just machines that are going to make it easier for people to do their jobs or that allow them to focus on more creative Endeavors I think I think the job market for sure will be evolving in the future but what I think it's important is that if we evolve all the privileged jobs and we don't forget to also make sure the jobs fall the less privileged people also evolve so that they can have much more meaningful lives perhaps spend time more time with their families and of course with this um I understand that this is very utopian and and I'm not touching the questions of Economics or motivation psychological factors and so on but these are also the things that we need to work on all the time it's really beautiful Nando um I would like to ask you one one last question um I know your path in terms of PhD at Cambridge postdoc at Berkeley professor at UBC in Oxford and then your own startup ending up at deepmind slash Google um but before then what what was the original thing that got you excited about artificial intelligence I was just lucky um like I think a lot of things in life a lot decisions you made but just you find yourself in a context and you make the best out of that context but you are often when you I couldn't say that I plan things very far ahead maybe maybe I plan to do a degree and then I knew that that was gonna last at least four years so um but quite often you encounter uh yourself in a situation in life whether it's a challenge or an opportunity and you have to identify uh what if yes what's what what you should do I was lucky enough that when I was in my third you know an undergrad um there was a project on neural networks and my control professor professor McLeod the diversity of the advanced thread and I said quality so I went to the library first um at first I thought oh no I don't want to do this it's going to be one of those PC VIP protocols things I'm going to hit this course but it was a completely different kind of networks sort of you open the book and there's a picture of a brain and I think that's it I got hooked there were a few books in the library there and I just couldn't stop reading about it and then I had to go in Matlab and Implement wine back prop and yeah I'm back from almost 30 years ago yeah right exposure during your undergrad studies I was lucky that I had that opportunity well then I feel really fortunate you had the time to come on to the podcast really enjoyed the conversation thanks so much for making the time thank you all so much Peter and I love your podcast and it's been a pleasure to listen to it uh you know through the years especially during the pandemic and third it was always a good companion it's it's a wonderful job that you do it's um yeah it's it really brings what we do to so many people thank you Nana really appreciate it

Channel: The Robot Brains Podcast

Views: 3,133

Keywords: The Robot Brains Podcast, Podcast, AI, Robots, Robotics, Artificial Intelligence

Id: dKvYRgOifFA

Length: 75min 32sec (4532 seconds)

Published: Wed Apr 26 2023