Jeremy Howard: Deep Learning Frameworks - TensorFlow, PyTorch, fast.ai | AI Podcast Clips

the from the perspective of deep learning frameworks you work with fast AI tickler this framework and pi torch intensive flow what are the strengths of each platform your perspective so in terms of what we've done our research on and taught in our course we started with Theano and care us and then we switch to tensor flow and care us and then we switched to PI torch and then we switched to PI torch and fast AI and that that kind of reflects a growth and development of the ecosystem of dig learning libraries siano intensive flow were great but we're much harder to teach and do research and development on because they define what's called a computational graph upfront less data graph where you basically have to say here are all the things that I'm going to eventually do in my model and then later on you say ok do those things with this data and you can't like debug them you can't do them step-by-step you can't program them interactively in a Jupiter notebook and so forth pi torch was not the first 4pi torch was certainly the there's the strongest entrant to come along and say let's not do it that way let's just use normal Python and everything you know about in Python is just going to work and we'll figure out how to make that run on the GPU as in when and necessary that turned out to be a huge a huge leap in terms of what we could do with our research and what we could do with our teaching and because it wasn't limiting yeah I mean it was critical for us for something like dawn Bench to be able to rapidly try things it's just so much harder to be a researcher and practitioner when you have to do everything up front and you can inspect it problem with pay torch is it's not at all accessible to newcomers because you have to like write your own training loop and manage the gradients and all this stuff and it's also like not great for researchers because you're spending your time dealing with all this boilerplate and overhead rather than thinking about your algorithm so we ended up writing this very multi-layered API that at the top level you can train a state-of-the-art neural network in three lines of code and which kind of talks to an API which talks from OPI which talks from API which like you can deep dive into at any level and get progressively closer to the Machine kind of levels of control and this is the first day a library that's been critical for us and for our students and for lots of people that have one big learning competitions with it and written academic papers with it it's made a big difference we're still limited though by Python and particularly this problem with things like recurrent neural Nets a where you just can't change things unless you accept it going so slowly that it's impractical so in the latest incarnation of the course and with some of the research risked out now starting to do we're starting to do stuff some stuff in Swift I think we're three years away from that being super practical but I'm in no hurry I'm very happy to invest the time to get there but you know with with that we actually already have a nascent version of the first AI library for vision running on switch 19 so far because a Python for tensorflow is not going to cut it it's just a disaster what they did was they tried to replicate the bits that people were saying they like about a torch the is kind of interactive computation but they didn't actually change their foundational runtime components so they kind of added this like syntax sugar they called TF eager tons of logging in which makes it look a lot like PI torch but it's 10 times slower than PI torch to actually hmm do a step so because they didn't invest the time and like retooling the foundations cuz their code base is so horribly company yeah I think it's probably very difficult to do that kind of rejoin yeah well particularly the way tensorflow was written it's written by a lot of people very quickly in a very disorganized way so like when you actually look in the code as I do often I'm always just like oh god what were they thinking it's just it's pretty awful so I'm really extremely negative about the potential future if it by the flaws of the FET Swift for tensorflow can be a different beast altogether it can be like it can basically be a layer on top of M lar that takes advantage of you know all the great compiler stuff that Swift builds on with LLVM and yeah it could be a think it will be that's what you fantastic well you inspired me to try Evan Roo truly felt the pain of tensorflow 2.0 Python it's fine by me but yeah but it does the job if you're using like predefined things that somebody's already written but if you actually compare you know like I've had to do because I've been having to do a lot of stuff with 10 so far recently you actually compare like okay I want to write something from scratch yeah like I just keep finding is like oh it's running ten times slower than pi torch so is the biggest cost let's throw running time out the window how long it takes you to program that's not too different now thanks to chance to flow eager that's not too different but because because so many things take so long to run yeah you wouldn't run it at ten times slower like you just go like oh this is taking so long yeah and also there's a lot of things that are just less programmable like TF data which is the way data processing works and tensor flow is just this big mess it's incredibly inefficient and they kind of had to write it that way because of the TPU problems they described earlier so I just you know I just feel like they've got this huge technical debt which they're not gonna solve without starting from scratch so here's an interesting question and if there's a new student starting today what would you recommend they use well I mean we obviously recommend fast AI and pi torch because we teach new students and that's what we teach with so we would very strongly recommend that because it will let you get on top of the concepts much more quickly so then you'll become an extra and you'll also learn the actual state-of-the-art techniques you know so you actually get world-class results honestly it doesn't much matter what library you learn because switching from China to MX net to tensor flow to PI torch is gonna be a couple of days work as long as you understand the foundations well but you think we'll Swift creep in there as a thing that people start using not for a few years particularly because like Swift has no data science community libraries Oh basil wing and the Swift community has a total lack of appreciation and understanding of numeric computing so like they keep on making stupid decisions you know for years they've just done dumb things around performance and prioritization that's clearly changing now because the developer of chris Christie at developer of Swift Chris Latner is working at Google on Swift Potenza flows so like that's that's a priority it'll be interesting to see what happens with Apple because like Apple hasn't shown any sign of caring about numeric programming in Swift so I mean hopefully they'll get off their ass and start appreciating this because currently all of their low-level libraries are not written in Swift they're not particularly swifty at all stuff like ml they're really pretty rubbish so yeah so there's a long way to go but at least one nice thing is that Swift for tensorflow can actually directly use Python code and Python libraries in a literally the entire lesson one notebook a first AI runs in Swift right now in Python mode so that's that's a nice intermediate thing you

Channel: Lex Fridman

Views: 82,439

Keywords: deep learning, tensorflow, pytorch, fastai, python, theano, machine learning, deep learning frameworks, swift, jeremy howard, artificial intelligence, ai, ai podcast, artificial intelligence podcast, lex clips, lex fridman, lex podcast, lex mit, lex ai, mit ai, ai podcast clips, ai clips

Id: XHyASP49ses

Length: 9min 1sec (541 seconds)

Published: Sun Oct 06 2019