Jeremy Howard - Creating delightful libraries and books with nbdev and fastdoc | JupyterCon 2020

hello and welcome to jupitercon 2020. i'm raeshma shaikh diversity chair for the conference i'm pleased to introduce our next keynote speaker jeremy howard jeremy is the co-founder of fastai and a researcher in residence at the university of san francisco as an open educator he's co-authored of free courses on deep learning that have reached hundreds of thousands of learners around the world including me he's also co-authored a open source library called fastai which i've personally used um earlier this year in 2020 jeremy co-founded the maths for all campaign to address the code of 19 pandemic and is the lead author on one of the most influential critical and widely read publications i'm presenting jeremy and his talk is entitled i love notebooks thank you i do i love notebooks um and i feel like that's always a dangerous thing to say particularly if there's somebody in the room who is a real programmer they always try to explain to me why i shouldn't like notebooks and why i should be liking something else instead like the s code or e-max or vim or whatever and there's always seems to be this assumption that the only reason i like notebooks is because i don't know about all these other much more um impressive and professional things now the truth is i've been programming every day for well over 30 years i've used all of those things extensively and i still like notebooks very very much in fact i've written a lot of things with notebooks i found it quite an effective way to build stuff that people seem to like including a number of software libraries uh such as uh the fast ai library which i built with uh silver gojo uh silver and i wrote all of fast ai in jupyter notebooks and um people seem to really like it um we're pretty proud of it of what we've built um one of the nice things about um what you see with fast ai is it's got a beautiful documentation and in the documentation you can click on at the top of any page there's an opening collab button and bang you're now not just reading documentation but you're interacting with it so you can actually fiddle around with the things that you were just reading about there's a big difference between the kind of dead tree experience of reading static content and the interactive experience of actually working with live objects and so it's really cool that we can do that thanks to jupiter notebook the trick to turning notebooks into really great software projects is something called mbdev uh nbdev is something that silva and i wrote that creates your python modules for you um it exports changes from if you use an ide your text editor from that back to the notebook it creates searchable documentation it creates installers it does testing it handles ci it handles uh version control and so forth i'll talk more about that um later um the experience of developing with jupiter notebook a library um is is just so different to anything that i've used before compared to vs code or visual studio or bim or amax or whatever you have everything together so for example here is a web server that i built in a notebook and it's a very simple little web server that inherits from python standard library web server and you can see how in the notebook i have a conversation with the reader which initially is a conversation with myself as i'm learning about how python's http server works and that exploration ends up being a part of the artifact i generate the documentation and it ends up being the actual code so it's a really different experience um hamill hussein from github has described this experience better than anybody else i've heard he said that with his experience of nbdev and jupiter notebooks the test stocks and code are part of the same context and are co-located there is no other programming environment that exists like this that i'm aware of you can even make notes to yourself about why something works the way it does very easily while you're writing the code and it isn't an afterthought this is fundamentally why i have a problem working in anything besides nbdev because not only does it make the code more approachable to others but forcing you to write docs actually forces you to think about the code more in my personal projects it is mv dev i often reflect my code to be simpler and better after forcing myself to explain it this is powerful i find exactly the same thing the process of interacting uh with the live code and thinking about how to explain it and how to test it all in the same all in the kind of the same place at the same time it's a very different experience um silver and i also wrote a book which people seem to like the entire book was actually also written in jupiter notebook and it was published uh using something called fast stock which is a layer on top of jupiter notebook to create actual publication quality books and folks like peter norvig and eric topple and many others have said they really like the book and we like it too we really think it ends up looking beautiful and the nice thing is that the entire book is available for free on github also in jupiter notebook format so you can actually interact with it every line of code everything you see here was generated by a notebook and you can see how to create it yourself how to work with it yourself experiment it with it yourself and fast stock is literally a case of pip install or condor install and then you write a single command fast dot convert all and it spits out your book i've also built a number of courses which have been very popular people seem to really like them and the entire all the courses are entirely taught within jupiter notebooks and it's not just that people like them they really work a lot of people that have taken these courses have gone on to present papers at places like europe's they've gone on to work at places like open ai and google it's really been a very successful course and our students tell us that the fact that it's taught through jupiter notebooks is one of the things they really like about it notebooks support a different kind of programming called literate programming letter programming goes back a long time to the famous professor donald knuth who described it as a methodology that combines a programming language with a documentation language thereby making programs more robust more portable more easily maintained and arguably and i feel this way more fun to write the idea is to treat a program as a piece of literature addressed to human beings rather than a computer and you can see that example in the in the little web server example i just showed you it really is describing what you're doing kind of to your future self to to you know to the computer as you're building it to your colleagues it's a really nice experience and i actually find it goes beyond literate programming what i call exploratory programming so for example uh dev is written in nvdiv and nbdev um as you read the source code this is the source code to envy dev it's actually describing and showing my expiration of well what on earth is actually a notebook let's try opening it oh it's json let's have a look at the json okay how do we convert that into a format that we can execute oh here's a little program that does that let's try it out okay now let's package that up into a function and so forth and so the process of exploration is critically important and particularly for a researcher for a scientist we're all taught at school that the scientists journal is their most important thing it's the journal of everything they've tried and what happens and with usual programming in an ide or a text editor you lose all that expiration you end up with just the final artifact so for the people who want to understand the code you wrote and the apis you're using and the services you're using you have to find another way to teach them or they have to find another way to find out um anybody who wants to use your library they're not being taken along on that journey with you so i find exploratory programming really powerful for developing better code more quickly but also for other people coming on board and immediately seeing what am i doing why am i doing it there are the tests right there there are pictures of the outputs right there that works really well so in the end notebooks give us a live voting environment and live actually live coding working with the code as objects it's so much better than a dead coding environment where it's just a text file live coding goes back a long way uh it goes back at least to small talk which back in the 80s and even the 70s provided an entire live coding environment where you were interacting with the code uh kind of in real time and in fact some people have done some really cool stuff with live coding for example there are actually people who have done performances using live coding here's an example of volume up here's an example of some actual live coding audio performance let's check it out so as you can see there he was actually working with the code and as he worked with the code the music was happening and so isn't that a great way to actually build something that you're listening to it you're manipulating it here's some of the most amazing live coding i've seen this is by an extraordinary researcher and developer named brett victor bounce off my turtle pause the game now hit this button here which shows my guy's trail so now i can see where he's been when i rewind this trail in front of him is where he's going to be this is his future but when i change the code i change his future so i can find exactly the value i need so when i hit play you subscribe [Applause] how often have you written code and have people go whoa and clap as you do so i mean this is pretty amazing and this is what we should be aspiring to an environment where whatever you're trying to build you can interact with that environment as part of the process of coding to build the thing you just build the thing you want it shouldn't be a case of jumping over to a text file and then back over to a compiler and then over back over to a bunch of tests this is what programming should feel like so brett's work was extremely influential and amongst other things that inspired chris latmer to build xcode playgrounds and swift playgrounds where you can actually see as something's running how many times is it running what are the outputs of it what does it print out in each line super super cool so people have really been pushing the envelope around live coding and i am so so proud that chris lattner who built this as well as llvm as well as swift um told us that he thinks that i really do think that nbdev is a huge step forward for programming environments so for chris to say that was really really exciting for me but this is not how real code is code we're told and in practice this is not how real code is send your code here's an example of some actual live coding in a more traditional environment we can see over here just a plain text file inside an editor and down here we have ipython a line-oriented ripple where your only ability to interact with history was basically to hit up arrow to try and find something you said before and hope doing it again looks the same and so here's what the experience of programming uh in this looks like so we had down here we tried to find the last thing we ran oh that was wrong we go back we try it again oh no we've got an assertion now we have to go somewhere else we have to try to figure out what's going on here that we have to type something back into the rebel now the rebel state is kind of in this weird situation where it's a weird mix of things then we go back into the text file and we try to change that and hope that's going to work and then run it again and it still doesn't work it's like watching this just feels like ah that's that's the way i used to code and you can see this expression here it's like ah that's my feeling of coding in that kind of environment we should be able to do so much better this idea that real programmers program like that it really reminds me of how people used to write text files they used to use ed a line-oriented repo for editing text files it feels a bit like oh real programmers don't use notebooks real programmers use ed or ipython and i really think that we can we can move beyond that but the line oriented ripple has a history there's a reason we have line-oriented rebels and the reason is because this is how we used to code see that's what we used to do when we were coding you know back in the 70s and before we didn't have terminals um we didn't have the ability to show graphics and actually manipulate things like so the best we could do is these line-oriented things and somehow so many people are still stuck with them now of course there is a place for line-oriented rebels but that doesn't mean they should be the the beginning and end in how we interact with code also these kinds of dead coding environments like we saw uh in that vs code live coding example um before um tend to lead to errors for example a book i was looking at the errata for a book that did not use notebooks like most don't but instead there was a separation between the artifact the book and the writing of it and the errata are full of things saying uh no the code actually doesn't work the code in the book is not actually code that gives the right answer and in some cases it's even saying oh there's actually a code repo for the book that has something that works but the book itself has something that doesn't work so this you know for for us when we wrote our book all the outputs in the book actually ran from the actual code in the book because the book is the code the code is the book it all comes out of notebooks so these are some of the issues i find with with working with traditional environments that the notebooks just completely avoid so why are so many people against notebooks in fact there was a talk here at jupitercon two years ago i don't like notebooks and it's actually by the person who i just showed you doing that coding in the s code it's by the person who actually wrote the book with all that code that doesn't actually work a guy called joel grouse and joel actually presented a really really funny talk a really really compelling talk but a really really wrong talk and so that was really unfortunate so i feel like in order to describe how to code better in notebooks i first of all have to explain why it's actually a good idea to and part of that is every time i say to somebody i like notebooks they always say oh haven't you seen that talk about how you shouldn't like notebooks so in order to describe some of the reasons this talk is wrong i'm going to show some of the slides from it and so you know which slides are from the talk i'm going to put this little picture down here so the ones with a little picture down here are slides from the top um so i was a bit concerned when i started looking at that talk because uh as joel said himself in a kind of a self-deprecating way he's not a notebook expert um which is fine um but when he first said on twitter as he also pointed out in his talk uh that he was going to talk about uh how he doesn't like notebooks i thought well that sounds like a bad idea when somebody who is not a notebook expert describes you know how you know he doesn't like notebooks when i know a lot of people actually are more productive and more effective by using notebooks that sounds like a bad idea um my feedback was not very successful and so i was pretty concerned that i knew he was a great communicator super funny guy and i thought he might write something that convinces a lot of people to not use notebooks and he might express a lot of opinions despite his lack of expertise in the field and i might have to come back to jupitercon a couple of years later and try to correct the record and well here we are here's what he said he that notebooks discourage good habits notebooks encourage bad habits notebooks encourage bad processes notebooks hinder reproducible and extensible science notebooks are a recipe for football affected code notebooks make it easy to teach poorly notebooks make it hard for me to teach well well if all of these things were true then i must be some kind of genius to have built so many things that people like despite notebooks being so bad or maybe the problem isn't the notebooks maybe the problem was a lack of understanding of what notebooks are and how to use them correctly and appropriately well luckily joel told us about things he doesn't like and about notebooks and these are things that i've heard lots of other people real programmers who also were not experts in the book say as well so i think it'd be a good idea to dig into them and understand them so one of the concerns i've heard a lot is this idea that notebooks have state that's easy to screw up and difficult to reason about and and even that there's this like incredibly complex thing about like oh how do you run each cell exactly once in order um i gotta say i don't get it there's a button that does exactly that um having said that i never use that button myself because nbdev runs every one of your notebooks for you in order and every one of the cells um in order and make sure that they all work so um for me um you know i find that the interactive ability to manipulate that environment and that code and go back and forth is critical to the exploration it's critical to being able to understand what's going on and what happens if i change something and what happens if i put it back again you know the the state is such a critical part of the actual experience of live coding rather than the dead coding experience of just having a text editor that just runs it top to bottom every time but it'd be fine if we could just have the whole program in our head and just throw it down there and run it but in real life we need to learn about how the apis work we need to learn about how the services we're using work or in my case a lot of people watching this case we're actually doing research we're actually trying to figure out how does this kind of neural network actually behave or what actually is going on in this microscopy image or so forth um there's also this concern of like well how do i copy and paste stuff out of a notebook into slack or a pull request or whatever and this is where this kind of deep misunderstanding is because that's what you do with dead code environments but in live code environments you can go so much better this is what a pr to our fast ai repo looks like here's a button you can click to take you to a collab notebook reproducing the behavior and i click the button and i can now run it and see exactly the behavior that this um uh person filing this issue has given me and so isn't this so much more powerful than copying and pasting some text when you copy and paste some text how do you get into that exact same environment how do you go through the process of like pulling out the outputs or the inputs the entire thing is so much less convenient than literally clicking on a link and these links are so easy to create if you've got something in github you can literally use a chrome extension to immediately turn that into a link that you can put in a pr or an issue or whatever and poor requests just look so much better with notebooks thanks to review nb when i get pull requests is an example from an actual pull request and one of the things that this person did was they changed some of the numbers in the scales here and so i'd love to know how does that change the what the documentation is going to look like well of course the code is the documentation with notebooks in mbdev and so i can literally see the outputs the green tells me they've changed and i can see the pros and it's all here in one place so otherwise i would have to have taken this pr checked it out run some things separately had a look it would have taken so much longer i don't even know i would have bothered frankly so you know there's other ways to share notebooks as well again you don't just share a cup copy and paste of some text because notebooks can have like this one pictures so there's a button you click which actually creates a gist literally one button and away we go another claim is that notebooks make it harder to um to use other people's code and i've got to say i've found the absolute opposite of this i find notebooks mean i can actually run things particularly because so often now people are sharing notebooks and things like collab so for example our entire book the o'reilly book is all available every chapter you can click a button it opens it on colab and the first cell is a cell that sets up the exact environment that you need and you start running it so we don't have any issues with installation we don't have to have something saying here's how to install all the stuff um to work with this book which you know otherwise would be quite complex it's got pie torch it's got the fast ai library it's got quite a few libraries that it demonstrates and uses in the book but we can do it all in a live environment where the reader is not just a reader but they're actually working with the code i've also heard these claims that that there's some conflict between good software that there's some conflict between good software engineering and workbooks this idea that there is a set of rules for software engineering that hey you can't follow if you're on using notebooks and b that you should use the exact same rules it doesn't make sense there are very different ways of writing software depending on what kind of software you're writing and what kind of environment you're writing it in so these rules we're told to follow there's certainly no reason the vast majority of time we couldn't follow them in notebooks but very often they totally misunderstand the pros and cons of the notebook environment and the rules don't always make sense there's also this claim that maybe notebooks reduce the ability to have modularity so here's an example from some of joel's code which is claiming is beautifully modular and that we couldn't do this in notebooks um this just isn't true fastai is a library written entirely in both books so where the modularity is so good that there's actually a peer-reviewed journal paper about about its layout api there's no tension between using notebooks and having modularity particularly when you use nbdev which can turn every single notebook into a module for you same with testability in fact testability the actual practice of testing is so much better in a notebook than in a dead coding environment because in a notebook the tests are literally there next to the code that's being tested and next to the explanation of what's going on so it really forces you to think through what is this test showing how do i describe this properly and if somebody then edits your code they can see the context and the tests right there with it so they don't have to jump backwards and forwards they can immediately understand like oh that's why this code was like this or maybe it's not lucky after all or maybe oh maybe i could show this back in some different way and now i can immediately add a test to demonstrate the behavior to ensure there's no regression in the future and so forth so notebooks are a great place to point tests and then of course those tests can end up in continuous integration and with nbdev that actually happens automatically all of the tests are in notebooks are run for you anytime you push something to your reader the other claim um made in joel's talk was that notebooks are actually worse for teaching and learning which i again it's it's against everything of my own experience um the the truth is that people find vast ai's courses a really great learning experience one of the things it just reflects a bit of imagination to think about like what is going to be helpful for the student for the student and one of the things we do in our book is we provide two versions of every notebook one version that contains all the pros and all of the code outputs and so forth and we also provide this version which is automatically generated which deletes every single output and every single piece of um pros except for headings so the first assignment for every notebook is to go through we call it the clean notebook go through the clean notebook and before you run each line think about why is it there and what is it going to do before you run it now we can't force people to do this we can't force people to learn um but it's a pretty good way for those that do want to learn um to have the opportunity to do so and then we tell people if the output of a cell is not what you expected go back and try some different inputs because obviously there was something about the process you didn't understand so go back and try some different inputs to understand how this code behaves and we find that people really appreciate this so it just requires some imagination and some empathy with a student to think about how best to present things a way that's going to support their learning text editors are amazing and vs code in particular which is the one that joel demonstrated in the live coding i showed early on and and during his talk they really are very impressive pieces of technology now there's claim that notebooks are less helpful than my text editor it doesn't make sense in fact truly live code must always provide better information than your text editor for example here's some code in jupiter um pretty weird piece of code i'm just trying to demonstrate something here this is obviously going to return a string because this is going to return something this is a working url so in jupyter uh the autocomplete correctly shows me the things that a string can do the s code not so much because vs code can't actually run the code and know what's going to happen so it incorrectly guessed that it's going to contain a number and it doesn't contain a number and this is this there's no way for vs code or any editor to give you the right answer here without actually becoming a notebook without actually running each cell and having you decide to run itself and it can't do that automatically right because it doesn't know which ones are dangerous it has no way to know ahead of time what a cell or what a piece of code is going to do so there's a fundamental disconnect between truly live environments and dead editor environments despite the fact that vs code tries so hard to do as much as it can for you so joel said that we could win him over to the notebook world uh by providing ide style auto complete well in fact um nowadays jupiter uses basically the same language server that um vs code does and so as you can see um even if you haven't run the previous cell it does a great job of guessing um that this is well this is a string it doesn't have to guess it's in the type annotation it knows this is going to be a file because it knows it open returns and so forth so really a notebook is going to give you an experience that is a superset of of that kind of intelligent auto-complete suggestions real-time type checklist linking with linting we're told we need to give joel for him to be interested well okay again um type checking in a live environment is always going to be more correct than in a dead editor environment here is about a dozen lines of code for the typed implementation in fast core you add that decorator to any function and it type checks it in real time right so because we're actually running real code not just making guesses this type checking is actually correct it actually makes sure that this really is what's happening and it's incredibly simple it's about a dozen lines of code compare that to what's required to do type checking in for example my pi which is the main uh type checking for non-live coding environments it is over a hundred thousand lines of code c code and c plus plus code and python code and it can never be correct it can never actually be correct because it can't actually run your code and actually know what's going on they can just make a guess and so often you have to modify your code to please my pi but that is not how these tools ought to work these tools should be helping us right whereas if you run if you use the typed decorator then it's working with you it's looking at your code at runtime which is how dynamic languages like python are designed for work that's the only way they can work anything like my pie by definition has to be a hack and it has to be an incomplete guess that you have to find ways to make it happy and then joel said he's looking for a better story around dependency management again i've heard this claim from many real coders that there's a problem with notebooks in understanding dependencies which i just don't get um you can use exactly the same dependency management approaches that you can in any python project and in fact with with nbdev it's literally a case of writing in a text file what your requirements are and that's all automatically handled for you and i think this is the final one he said is he wants first class support for refactoring code out of notebooks into modules couldn't agree more and of course that's exactly what mvdev provides so we start out with um this is actually the source code for mbdiv itself in a notebook and this is nbdev creating its own nbdev module in a file so yeah that's certainly important so all those things exist but joel said the reality is you're not going to provide me with these things and i'm not going to switch to notebooks so so be it i would like to think that the fact that we provided all those things might mean other people are interested in switching to notebooks if they're not already using them or if you are using notebooks you might consider using these things if you are interested let me tell you about what these things are the key to them is nbdiv nbdev is a project that we built which will create your python project create your python libraries um it will take uh everything out of your notebooks and put them into standard python text files you can change the text files directly in your editor or ide and it will update your notebooks for you it will create documentation it will create installers it will run your tests in parallel nice and fast it will handle your continuous integration and it will handle your version control let's look at each one of these in turn so here's an example of the code for nbdev as it exists in notebooks and so you can see the way i code and there are lots of ways of doing this this is my personal preference the way i offer encode is i'll start out by creating a class that kind of just has an in it and i'll start writing the documentation about what this class is for and it doesn't do anything yet right and so then i'll start to just say like okay so how do we instantiate this right so we've now shown how to instance yet okay so what does it do well a notebook processor processes special kinds of comments so now i start to build out some methods and one of the tricks here you'll see is i use the app patch decorator from fastcar which will insert this method into this type so you can absolutely write uh the entire class in one cell and then you can document it separately later there's a thing called show doc which will create documentation for a particular method at a particular time um or you can do it this way where you do it kind of piece by piece um i find some classes i do it in all in one cell some classes i separate them um in the end it's really up to you but this is a very handy way to gradually build up plus and this is actually key to donald knuth's original idea of literate programming which is you should be able to write stuff at the top that doesn't that that refers to things that don't even exist yet and that's totally fine because python does late binding right so you can have methods being called up here which we haven't even written yet because if you add them in later then that's no problem python absolutely is happy with that and the jupiter notebook will be happy with that as well um the ability to turn this into a library um comes out of the uh a single configuration file that you write uh so here is the nbdev configuration file itself then you just say what it's called and where does it live on github what's it about who wrote it stuff like that what are the requirements you can add console scripts stuff like that and the nice thing is now you just have all of this in one place in one file and it handles everything for you so there's so much less to learn you don't have to worry about pie pie versus conda this is what setup.pie is have version numbers in lots of different places and so forth it's just one file in one place and everything in nvdev that needs it knows to look in this one place and so amongst other things this gives you a two-way synchronization where again here is the source code for some of mdf and when i run nv dev build lib it will create this python file and when i run nvdiv update notebook it will pop the this file back into this notebook and you can see in the file it knows where each cell came from so it knows what to update um when you do so so this to say synchronization is nice i do all of my changes pretty much in the notebook but sometimes i might want to do a quick search and replace um you know using perl or something like that or there are some things that work better in a text editor and so i use a text editor for those and then sync it back to the notebook um so the process of going from notebook to module is done by i add a cell at the bottom of every one of my notebooks which imports notebook to script and runs it and so that way the notebook can export itself so anytime i say run all cells one of those cells is actually going to turn itself into a python module or at the command line nb dev build lib will do exactly the same thing and also with nbdev we provide a make file that knows when to run these things and handles it all for you there's a lot of little things that nba dev handles for you for example it creates the special under all variable in each module which is the thing that python uses to decide what to import so if we create a little script a little bash function that shows us all the stuff that gets imported and have a look at say fast fast core transform you'll see that what's imported by default is a nice small list of things it's just the things we wrote in nvdiv if we pick another um if we pick another project so not entirely at random let's pick the one that joel works on which is lnlp and we look at what's imported from one of their um modules json copy logging default dick eddie these are not things that were actually written in this these are parts of python but they didn't have it done to all right so so with nbdev we try to make sure that all of these important details are handled for you the documentation is really cool um here's the source code for the markdown cell that we put underneath the source code for this uh part of um fastai and you can see here that every one of these back tick sections has been turned into a hyperlink even those which are in different libraries um and so it's really nice that you don't have to think about how to hyperlink to the other parts of your documentation just put backticks around it and nbdev handles it all for you and you can even set up an nbd of things to say look in these other third-party libraries as well as part of that process because otherwise you know a lot of us just wouldn't bother with this but when it's this easy it really helps your users to find their way around what's going on in your library and the um the the documentation really ends up looking very nice and it's very clear you have um an automatically created uh set of um kind of a separate page for each of your modules it can be placed into separate parts of the menu you automatically get a table of contents for each one you can automatically get an open call up batch for each one as you can see you know you really end up with a a nice experience without you having to think about it right this is all done um for you um building documentation you can either run nbdev.org docs and it creates the docs for you right there in your repo and then in github you can just say enable github pages and you automatically get documentation or you can create a little action github actions job or something like that we've got a buildbox um action which will automatically build your documentation and host it somewhere else so we use both approaches in different places so um another nice little thing is that you have a special index notebook index.ipind um and here's the actual index notebook uh the top of it from the fastai library and we will automatically take that for you and generate your readme file um we will automatically take it for you and generate your documentation homepage we will automatically take that for you and create your pie pie description um so again there's all these kind of little things that you want to do to create a really kind of delightful experience for your users and doing this all by hand it's a lot to learn it's a lot to maintain it's a lot to think about so with nvdev it's all done for you um and then you want to make sure that people can install it using either pip or condo again that's all handled for you you just type make release and it will package it all up onto pi pi and onto the anaconda cloud for you so people can install it using conda or pip and again all the descriptions and everything like that is all put in there for you without you thinking about it so something like this fast web hook which is a little web server i mentioned it's just a tiny little utility thing that i've mainly built for myself um but because i created it with nbdev um i end up with something that that other people can use i end up with something that's installable and it's documented and um easy to maintain so i find that even for like simple little things i always build them with nb dev now and so everything ends up being a really nice clean project that other people can use version control is handled for you if there's a a merged problem which obviously happens a lot with notebooks nvdiv will automatically ignore any merge issues in outputs because you just can recreate those outputs and if there are actually differences in the cells themselves then nvdiv will actually change the notebook so you end up with a notebook level diff so you're always guaranteed you can open the notebook and you can actually see the changes and work with the cells directly which i find super super helpful testing in nbdev is um it becomes part of the documentation it's part of the process of writing the notebook and then when you run the test you can just run nb dev test nds and it will take each notebook and run the tests uh in parallel and that's super fast and super handy but by having it also in continuous integration which again is automatically done for you um even if you forget to run it it'll let you know if you've got any failing tests the documentation handles all the little things that you might want for example it's got very nice math support that works really well and is much faster than the mathjax library um and then bdev actually powers some other really cool things as well that powers fast dock that i already told you about which can let you create publication quality books such as our o'reilly book and fast pages which you might have heard hamlet hussain talk about this morning which is a really nice blogging platform that is built on top of nbdev and turns all of your notebooks into blog posts for you so using this approach you can use a real exploratory programming environment to to create all the things you want your documentation books blog posts libraries and you can do it in a way that your head is in one place you've got all your tests there you've got all of your pros there you've got all of your code there and you can bring other people along with the right as well lots and lots of people are using nbdev around the world now um and i hope that you might consider joining them um if you go to nbdev.fast.ai you can get started with it the whole thing is free and open source and it's ready for you to use so thanks very much for listening to my talk and i will check if they have any questions there is one question in the chat which is would love a few thoughts comparing mb dev and jupiter book sure um so um jupiter book is very different nb dev the more similar thing would be between jupiter book and fast stock so far stock um is the thing that creates publication quality books and uh jupiter book can do something pretty similar um jupiter book didn't exist when i started and silver and i started with fast stocks so we haven't really investigated it that closely it seems to do some similar things i know it does live in very different ways so i would say that people you know try both and see what works for you um jupiter book is not going to build your python projects for you or create installers or any of the stuff that nbdev does i would say one of the benefits of using the kind of nbdev ecosystem is that you know you have all of your configuration in one place there's just one thing to think about um you don't have all these different pieces working in different ways but i would say yeah try both and see what works for you we have a few minutes so if anybody else has any more questions you can post them either in mattermost or the q a and webex and we can take this opportunity to answer them so i do have a question so mb dev and fafsa and all the pages are is that documentation under fastai or is that under jupiter's documentation uh it's it's that each one's got its own documentation website that's how envy dev works so indeed documentation is at nbdev.fast.ai ais is at first ai dot fast at ai um fast courses at first chord are faster ai and so forth okay i see some questions coming in the chat does nbdev save global editor state something like an undo button um no uh for editing uh mb dev just uses jupiter directly so jupyter already has undo so you can hit the z button in jupiter and it will undo a delete of a cell uh control z when you're editing a cell will undo edits to the cell and it's really cool each cell has its own undo history in jupiter so if you press undo it'll actually undo the last edit to the cell that you're currently in which works really nicely okay and there's another question are there any other projects you would highlight that you think are good examples of using mbdev in practice um i wouldn't say i'm familiar with them actually so i don't i'm not sure okay and another question is with which kernels does mbdev work uh it just works with python at the moment um it would be really cool to add julia or our support to it wouldn't be too difficult to do but it's not something i've worked on myself yet sirens are allowed in new york um i don't see oh i see one more person typing a question in matter most i'll give them a chance to finish typing do you do you think one of the issues joel had was that most of what you showed is not shipped in jupiter by default um could be um i think you know the the issue at the end he said uh nobody's gonna build these things anyway and the fact is um people are building a lot of stuff with notebooks and on notebooks and i think i think the important thing is to have the kind of imagination to think about what's possible and to also kind of not have the hubris to think that the current tools that you've used for years are the best and think about like you know how do we actually make it so people can manipulate the code more directly um you know it's something which a lot of um uh folks like like sutherland and know victor have talked about for decades or even it's like really thinking about how do we best support the development process and that means kind of innovating um uh what i'd like to see more of this stuff built into jupiter so when you download it you can get free yeah maybe um although one of the cool things about jupiter is it's a platform that people can build on and so typing pick and store mvdev is not really a big request for people to do um so yeah i think that i think having everything in the box is a nice idea but it's fine to expect people to install stuff too okay and another question from hamel here how has nb dev changed to improve the way that people contribute to fast ai projects um so it's you know speaking of somebody who's open source for many decades um i so much more prefer working with nbdev projects i would never want to go back now i as i described as i showed the issues that people have or the prs that people have i can see the change to the outputs in the pr i can see the actual kind of collab links to see how things actually work and i also find that people people just have higher quality pull requests because they know the context of the code that they're looking at because although all the documentation is there in front of them the tests are there in front of them you know it's much less common to have a pr that doesn't have documentation for example because it's very obvious when you're working in the notebook that all the cells around the one you're in have markdown pros so you probably should as well so it certainly um improved for me as a maintainer in my my life okay we have another question do you have a preference of jupiter notebooks or jupiter lab and what are the pros and cons of each um for me jupiter notebook does everything that i need um so that's what i use um the the extra stuff that jupiter lab adds around kind of like um tabs and all this kind of stuff um i i use the native browser stuff for that i'm very familiar with the keyboard shortcuts for you know windows and so forth for tiling windows and switching between windows so i prefer to use the stuff that's built into my operating system and my browser for all that having said that i think jupiter lab has great potential um particularly with stuff like graphical debuggers that are starting to be built um and so you know maybe in the future there will be things in jupiter lab that that make me want to switch could i say i'm just one word of happiness after listening to your talk you know you've touched so many so many points that made me chuckle and just uh gave me um give me a a nice a nice reaction uh because i uh especially as an educator have felt the same things that you were talking about i have seen uh so many times in novice programmers a spark of discovery and a spark of joy with this thing that you said in the beginning of your talk experiencing notebooks is like having a conversation with myself that um is a a very powerful tool of connection that we can use between our thoughts and um and and the computer's thoughts if you will uh by executing uh code and uh and so i just want to say your talk made me happy i'm so glad you know i um you know iverson who built apl back in the late 50s and early 60s uh he won the turing award for his contributions and he wrote a really powerful paper called notation as a tool for thought in which you talked about how the notation we use to interact with a computer and to describe our algorithms actually can change and improve how we think and i actually think that the computing environments we work with can do that as well and this is something that ivan sutherland talked about a lot you know the inventor of the mouse um uh he you know he he showed and talked about how we should be pushing ourselves to think about better ways to interact with a computing device and i think the idea of like typing into a text editor as like brett victor showed with that fantastic game example we can go so much further we can do so much more and i really encourage people to to be bold and to think about like what's the next step beyond nbdev can you build something that makes nb dev look stupid and simplistic i would be thrilled and and um for me that goes back all the way to the first um ideas about computational thinking uh that seymour papert was about when he invented logo and he was thinking about young very young people uh learning to discover the world through computing it's a very good my daughter is now learning coding she's she's five and uh it's it is terrific and i've noticed that for kids the coding environments are much more live they're much more explorable so for example um scratch which daughters now using you literally click on objects and you attach behaviors and as you do it you see them move you hear them make sounds and this is what i think we should all be doing when we answer that's a a descendant of logo scratch coming from the same school yeah well um look uh rishima do you want to close um the session as we reach the end and stop the recording and yeah um i love goodbye yep i want to thank jeremy so much for his keynote it was uh it was wonderful to hear and great questions and answers um next for the participants if you want to go to the announcements channel in mattermost for instructions uh about the poster session that's what's coming up next thank you everybody

Channel: JupyterCon

Views: 3,444

Rating: 4.9402986 out of 5

Keywords:

Id: HKt19-GsA1I

Length: 59min 11sec (3551 seconds)

Published: Fri Nov 13 2020