Heroes of Deep Learning: Andrew Ng interviews Ian Goodfellow

hi-yah thanks a lot for joining us today thank you for inviting me actually I was glad to be here yeah one of the world's most visible deep learning researchers life asked me to share a bit about your personal story so how do you end up doing this work that you now do yeah that sounds great I guess I first became interested in machine learning right before I met you actually I've been working on neuroscience and my undergraduate adviser Jerry Kane a Stanford encouraged me to take your internet AI class I didn't know that okay so I had always thought that AI was a good idea but that in practice the main thing I knew that was happening was like game AI where people have a lot of hard-coded rules for non player characters and games to say different scripted lines at different points in time and then when I took you were injured AI class and you covered topics like linear regression and the bias and various variants decomposition of the error linear regression I started to realize that this is a real science and I could actually have a scientific career in AI rather than nursing great and then what happe well I came back an IT a two-year course I'm late I see great education so the really big turning point for me was while I was teeing that course one of the students my friend Ethan Dreyfus got interested in geoff hinton deep belief that paper and the two of us ended up building one of the first GPU CUDA based machines at Stanford in order to run Boulton machines in our spare time over winter break and at that point I started to have a very strong intuition that deep learning was the way to go in the future that a lot of the other algorithms I was working with like support vector machines didn't seem to have the right asymptotics that you add more training data and it gets lower or for the same amount of training data it's hard to make them perform a lot better by changing other settings and at that point I I started to focus on deep learning as much as possible indeed and remember Magette rain has very old GPU paper race acknowledges you for having done a lot already work yeah yeah as I was written using some of the missions that we built the first machine I built was just something that Ethan and I built that Ethan's mom's house that I would wait with our own money and then later we have money to build the first to agree for the Stanford lab wow that's great I never knew that story of Jason oh and then today one of the you know things that's really taken the deepening world by storm is you invention of Gans so how did you come up with that I just studying generative models for a long time sort of a Gans our way of doing generative modeling where you have a lot of training data and you'd like to learn to produce more examples that resemble the training data but but they're imaginary they've never been seen exactly in that form before there were several other ways of doing generative models that had been popular for several years before I had the idea for again and after I've been working on all those other methods throughout most of my PhD I knew a lot about the advantages and disadvantages of all the other frameworks like Bolton machines and sparse coding and all the other approaches that had been really popular for years I was looking for something how to avoid all of those disadvantages at the same time and then finally when I was arguing about 200 models with my friends in a bar something clicked into place and I started telling them you need to do this this and this and I swear it'll work and my friends didn't believe me that it would work I was supposed to be writing the deep learning text book at the time but I believed strongly enough that it would work that I went home and coded it up the same night that it worked so thank you one evening to implement the first version as I if I implemented it around midnight after going home from the bar where my friend's house is going-away party and the first version of it worked which is very very fortunate I didn't have to search for hyper parameters or anything it was just for me I read it somewhere where you had a deaf experience and that reaffirm your commitment to AI tell me that yeah I was I wasn't actually near deaf but I briefly thought that I was I had a very bad headache and some of the doctors thought I might have a brain hemorrhage and during the time that I was waiting for my MRI results to find out whether I had a brain hemorrhage or not I realized that most of the fact I was having worry about making sure that other people would eventually try out the research ideas that I had at the time in retrospect they're all pretty silly research ideas but at that point I realized that this was actually one of my highest priorities in life was carrying out my machine learning research work yeah that's great that when you thought you might be dying soon you're just thinking how we get the research done yeah yeah that that that that's commitment yes yeah so today you're still at the center of allow the activities with scans of generative atmosphere networks so tell me how you see the future of Gans right now Jen's are used for a lot of different things like so my supervised learning generating training data for other models and even simulating scientific experiments in principle all of these things could be done but other kinds of generative models so I think that games are at an important crossroads right now right now they work well some of the time but it can be more of an art than a science to really bring that performance out of them it was more or less how people felt about deep learning in general 10 years ago and back then we were using deep belief networks with bolts and machines as a building blocks they were very very finicky over time we switched to think like rectified linear units and Bachelor realization and deep learning became a lot more reliable if we can make games become as reliable as deep learning has become but I think we'll keep seeing games used in all the places they're used with much greater success if we aren't able to figure out how to stabilize games but I think their main contribution to the history of deep learning is that they will have shown people how to do all these tasks that involve generative modeling and eventually we will replace them with other forms of generative models so I spend maybe about 40% of my time right now working on stabilizing games he's cool oh and so just as a lot of people they join deep learning about ten years ago such as itself ended up being pioneers maybe the people they joined Gans today if it works out could end up the early pioneers yeah a lot of people already are early pioneers of games and I think if you wanted to give any kind of history of again so far you'd really need to mention other groups like indigo and Facebook and Berkeley for all the different things that they've done so in addition to all your research you also co-authored a book on these learnings oh that guy that's right with Joshua Benji oh and Aaron kohrville who were my PhD coat visors we wrote the first textbook on the modern version of deep learning and that has been very popular both in the English edition and the Chinese Edition we sold about I think around 70,000 copies total between those two languages and I've had a lot of feedback from students who said that they've learned a lot from it one thing that we did a little bit differently than some other books is we start with a very focused introduction to the kind of math that you need to do deep learning I think one thing that I got from your courses at Stanford is the linear algebra and probability are very important that people get excited about the machine learning algorithms but if you want to be a really excellent practitioner you've got to master the basic math that underlies the whole approach in the first place so we make sure to give a very focused presentation of the basics that's a start book that way you don't need to go ahead and learn all of linear algebra but you can get a very quick crash course in the piece of the linear algebra that are the most useful for deep learning so even someone whose math you know is real shaky you've ever seen in math for a few years we're going to start from the beginning of your book and get that background and get into deep learning all of the facts that you would need to know are there it would definitely take some focused efforts and practice that making use of them great if someone's really afraid of method it might be a bit of a painful experience but but if you're ready for the learning experience and you believe you can master it I think all the all the tools that you need are there as someone does work in designing for a long time I'd be curious if you look back over the years tell me about how about how you're thinking of AI and a deep learning has evolved over the years ten years ago I felt like as a community the biggest challenge in machine learning was just how to get it working for AI related tasks at all we have really good tools that we can use for simpler tasks where we wanted to recognize patterns in hand extracted features where a human designer could do a lot of the work by creating those features and then hand it off to the computer and that was really good for different things like predicting which adds the user would click on or different kinds of basic scientific analysis but we really struggled to do anything involving millions of pixels in an image or a raw audio waveform where the system hasn't built all of its understanding from scratch we finally got over the hurdle really thoroughly maybe five years ago and now we're at a point where there are so many different paths opens that someone who wants to get involved in AI may be the hardest problem they face is choosing which path they want to go down do you want to make reinforcement learning work as well as supervised learning works do you want to make unsupervised learning work as well as supervised works do you want to make sure that machine learning algorithms are fair and reflect biases that we prefer to avoid do you want to make sure that the societal issues surrounding AI work out well that we are able to make sure that a AI benefits everyone rather than causing social of people and trouble with lots of jobs I think right now this is really an amazing amount of different things it can be done both to prevent downsides from AI but also to make sure that we leverage all of the upsides that it offers us and so today there are a lot of people wanting to get into AI so what advice would you have for someone like that I think a lot of people that want to get into a I start thinking that they absolutely need to get a PhD or some other kind of credential like that I don't think that's actually a requirement anymore one way that you could get a lot of attention is to write good code and put it on github if you have an interesting project that solves the problem that someone working at the top lab one is itself once they find your github repositories they'll come find you and ask you to come work there a lot of the people that I've hired or recruited at opening I last year or at Google this year I first became interested in working with them because it's something that I saw that they released in open-source form on the internet writing papers and putting them in archives can also be good a lot of the time it's harder to reach the point where you have something polished enough to really be a new academic contribution to the scientific literature but you can often get to the point of having a useful software product much earlier so sort of you know Nietzsche book practices and materials and post like github and maybe on archive I think if you if you learn by reading the book it's really important to also work on a project at the same time to either choose some way of applying machine learning to an area that you're already interested in like if you're a field biologist and you want to deep-learning maybe you could use it to identify birds or if you don't have an idea for how you'd like to use machine learning in your own life you could pick something like making a tree view house numbers classifier where all the data sets are set up to make it very straightforward for you and that way you get to exercise all of the basic skills while you read the book or while you watch Coursera videos that explains the concepts to you so over the last couple years have also seen you do one will work on adversarial examples and tell us a bit about that yeah I think every searle examples are the beginning of new fields that I called machine learning security in the past we've seen computer security issues where attackers could fool a computer into running the run code that's called application level security and there's been attacks where people can fool a computer into believing that messages on a network come from somebody that is not actually who they says they say they are and that's called Network level security now we're starting to see that you can also fool machine learning algorithms into doing things they shouldn't even if the program running the machine learning algorithm is running the correct code even if the program running the machine learning algorithm knows who all the messages on the network really came from and I think it's important to build security into a new technology near the start of its development we found that it's very hard to build a working system first and then add security later so I am really excited about the idea that if we dive in and start anticipating security problems with machine learning now we can make sure that these algorithms are secure from the start instead of trying to patch it in richer actively years later thank you fellas great there's a lot about your story that I thought was fascinating and that despite having known you for years I didn't actually know so thank you for sharing all that oh very welcome thank you for inviting me here great shot thank you very well

Channel: Preserve Knowledge

Views: 120,398

Rating: 4.9639344 out of 5

Keywords: p vs np, probability, machine learning, ai, neural networks, data science, programming, statistics, math, mathematics, number theory, pi, terry tao, algebra, lecture, analysis, abstract algebra, computer science, professor, harvard, MIT, stanford, yale, prime, prime numbers, fields institute, hinton, deep learning, nips, CLVR, computer vision, AI, talk, LSTM, sutton, bengio, facebook, google, google brain, alpha go, ml, cousera, andrew ng, geoffrey hinton

Id: pWAc9B2zJS4

Length: 14min 56sec (896 seconds)

Published: Tue Aug 08 2017