Jupyter notebooks vs Python projects: Learn when when to use which | Ep 1

[Music] this is a very hotly debated topic i guess it's this idea of notebooks versus projects [Music] in this video when will and sebastian discuss notebooks they don't mean the three-hole punch kind they're talking about a jupiter notebook that looks like this notebooks are a form of interactive computing projects are a collection of python files that make up a python package like the example shown here if you learned machine learning and data science in a university class today most likely you were taught to use a notebook first but that's not really representative of what happens in the real world meaning when you go to industry or an actual company i think there is no right and wrong like there is always like a use case for every tool there is out there otherwise this tool would probably not exist personally i'm actually a big fan of notebooks but it depends of course what you use it for so we just want to talk about what are the differences between developing notebooks and when not to use notebooks [Music] so i personally use notebooks when i want to prototype an idea this happens a lot in research where you think of something and you're like maybe i should try this model so um i would get into a notebook very quickly and just like just put the code down as fast as possible to know if this is even feasible or not and then i think the other times that i use it for are really like analyzing a model or analyzing the results of something or interacting with it like maybe i want to run an input through a model to see what it is what are the gradients what are the vectors look like do some quick embeddings like some exploratory stuff as well so i'm curious when do you use notebooks a notebook is really yeah a scratch pad a notebook where you take notes and it's not something maybe that you take away as a final product in your software engineering but a notebook is still i think something fine that we can use for ourselves to uh yeah scratch out our ideas another thing i like jupiter notebooks for is really when i want to explain something to someone because jupiter notebooks they allow you to execute things um yeah incrementally so i can actually prepare my script i can have a notebook for a given concept i can have images in this notebook i can have my mathematical equations and some notes and i can really step through this incrementally which is yeah incredibly hard with a python script which yeah runs the whole code all at once if you're teaching a mathematical concept like optimization or just models themselves probably very useful because you want you want the students to be focused specifically on that on those ideas but realistically you don't want them to leave that class without knowing how to convert that into a project so that they can actually do meaningful work with it [Music] i encourage people like once once you are past that poc phase where you're like okay i get it this is going to work i would say convert to a project and make it into an actual python package because it's just going to make time to production and time to you doing something meaningful with it very fast what you find today is that a lot of companies are having problems taking those notebooks and turn them into a production system because they're not built for production these notebooks end up going to an engineering team you know you throw it over the fence and then you hope and pray that like a year later they roll this out and it takes like six months a year for that team to understand what's going on turn into something that's scalable as well so i think that's really where you start to have the issues with it and this includes even paper publishing like i mean how many of us go through the paper writing process we publish a paper it gets accepted and then what do you not want to do you don't want to clean up your code on open source it right so it's really hard to then have something for the next researcher when they want to reproduce your results or you want to have a baseline that's your paper yeah it's going to be really hard to do it in a notebook so it's really meaningful if you do plan to open source your code and a lot of academic conferences today require you to open source something or show some code yeah this is a good point you have to be also mindful of who is the audience of your code is this just for yourself is this part of a bigger project this is something you're going to share with collaborators and yeah you want to let's say reduce the burden uh on let's say uh your future self and also your collaborators try to think of use cases where things might break is a jupiter notebook the right right tool for example for um yeah debugging your code i mean it's a it's a great a scratch pad but also as you keep notes um for yourself let's say on a piece of paper this piece of paper is nothing but probably that you are going to share with collaborators you will have a more nicely written let's say a meeting note file on your computer that you are going to share the more polished version in the same sense you probably don't want to share your your notebooks i use notebooks a lot but also it doesn't mean i just use a notebook i have more like a hybrid approach [Music] in the beginning of a project i usually start with my notebooks but yeah the bigger the project becomes the more i move things from the notebook into python scripts sometimes i tie them together with my jupyter notebooks but it's an organic process so you develop a feeling doing some projects when to use which tool it's a learning curve no right or wrong start with a tool that feels right for you but if you notice there is some downside to it don't feel like you are stuck with this tool feel free to experiment [Music] again takeaway is there's no right or wrong answer really depends on your workflow if you're in a poc stage or you want to experiment some ideas notebook is not a bad idea and actually will help you move really quickly another use case for notebooks is when you want to analyze something or you want to understand what's going on or interact with your model or what you're working on once you know that you're going to evolve that and you're likely going to collaborate with people or you're in an industry or you're going to publish that open source then it's probably a good idea to go ahead and migrate it into a project itself one tool is not better than the other it's just different tools are better for different things just be mindful of the task and the problem you're going to solve and also yeah think about your future self making life easier for you for your collaborators and pick the right tool for the task thank you for taking time out of your day to hang out with us i hope it was fun i had fun so we're going to be doing a lot more of these so stay tuned and hope to see you again [Music]

Channel: Lightning AI

Views: 11,503

Keywords: deep learning, pytorch-lightning

Id: JGnoTN1OnWY

Length: 6min 21sec (381 seconds)

Published: Mon May 23 2022