JupyterLab and JupyterHub - Perfect Together | Carol Willing @ PyBay2018

[Music] hi everybody how many folks have used the notebooks and love them how many aren't quite sold on the notebooks benefits or usage and how many have never used the notebooks and don't know okay so a few ok cool well for those of you that haven't used it at all I'm gonna go kind of fast but I think you guys will all keep up and I hope to answer the question for those of you that don't quite get where the notebook fits in to your workflow hopefully I will give you some ideas of what might be useful to you ok so this is really a talk for beginners but it has a lot of content that's new with Jupiter lab as well as Jupiter hot Jupiter lab is going to be the next generation of the notebook so different user interface a little more composable more IDE like but still has some of the simplicity that the original classic notebook has Jupiter hub is a hosted version of the notebooks which you can use in large classrooms like it Berkeley or Cal Poly where I'm at we teach students so that they don't have to download stuff to their own computers or have the latest and greatest computer they're just running through a web browser ok so i'm carol willing probably the only useful thing to know about me is i'm passionate about education and helping people reach their potential through using software and software tools and one of my things that drew me to Jupiter as well as Python was how much you could get done in a little bit of code because we have such rich libraries and how readable those codes are ok so the notebook itself sort of started in the scientific community as Travis had said and Fernando Perez and bright Granger got together with men right Kelly and came up with this interactive replacement or user interface for the repple the ipython rebel and the notebook lets you have code prose visualizations and a way to communicate ideas so if you're working on a scientific research project you might have something that you run one day something that you run another day how do you keep track of it the old school was lab notebooks hard copies some of them are still around and now you're seeing more electronic ones but what are the benefits of the notebooks in science is the ability to reproduce your research and have other people reproduce your research and be able to share and collaborate with those notebooks so before I get to Jupiter lab I want to talk a little bit about the classic notebook and as I said it has the combination of code prose images visualizations you can link in YouTube videos which really makes it a powerful teaching tool as well as a powerful prototyping tool and a tool for doing data science which I'm sure quite a few of you in here are doing but also people in let's say Japan at a university they've used it to communicate their operations workflows to their differences admins if there's a site crisis they have sort of their steps of how they recover and what they do in notebooks so it's not solely data science and if you think of it as more an interactive book or an interactive document where you can communicate different things you can execute code and provide guidance to somebody and share it that's about right and over 50 something languages now I think it's over a hundred languages so for those of you that our new to the notebook it is a fairly simple user interface in the classic notebook very linear their cells as you go down you put code or prose or in those cells and then hit shift enter and something magically will happen hopefully not a trace back hopefully like a lovely visualization or something meaningful so project Jupiter where are we today there are millions of users there's over I think I heard 2.6 million notebooks now on github and we were honored as sort of the de facto standard by the ACM and one the software systems award which other folks like UNIX and the web browser and cool stuff like that has won but you can see it's a variety of people that actually use the notebook and I think that's something to keep in mind [Music] we talked about reproducible science for those of you that are interested in space or black holes LIGO there was when there was a big you know information about it there's actually a LIGO tutorial that you can just click a button on binder which is a service that essentially lets you take a github URL and put it in a field hit launch and it will launch an ephemeral container out there on the web with all your dependencies and give you the notebook interface and we're the Jupiter lab interface and let you work on it there you can download stuff and then when your session ends everything goes away notebooks allow open journalism when I was in the Philippines I saw people in a news organization communicating between their support people and their sales people and their product people through notebooks and what they were finding was they were able to get more information that was useful to different team and cross pollinate faster than traditional code-based and memo based communication folks like O'Reilly has done used it taught in author interactive books for those of you that are interested in economics there's a project called quant econ that has phenomenal resources on Python as well as julia on how you know essentially a graduate level class in economics really well done so if we look at the notebook and Jupiter lab in particular we have some different building blocks for interactive computing we have file browsers we have code we have consoles and then the notebook interactive interface as well and many people need to use all of them in their workflow and with Jupiter lab we give the ability to have multiple windows that open at once have more of a traditional file browser launcher and something that you're more familiar with like if you're using atom or vs code or vim pick your favorite editor and as well as getting the benefits of the notebook and we'll go a little deeper into Jupiter lab if it goes forward okay so Jupiter lab its extensible by design it uses modern JavaScript for the front end and there's a lot of separation between functionality and how things are displayed and because of the way it is architected it is very straightforward for third parties to extend what you see within jupiter loud just like in vs code you have different plugins that you can you know have functionality for like a linter or something else you will see over time different things you might see biology or anted plugins you might see machine learning plugins and really tables of contents things like that Jupiter lab today can be found on github and it is an open source project about 100 contributors almost three years of work and a lot of releases primarily because its components are released in different stages and you know between the JavaScript and the Python and really it says what's currently in beta but it's practically for users it's a one dot IO deployment at this point it's very stable for those of you that are developing those third-party applications the api's are pretty stable I don't work directly on Jupiter labs so I can't say that they're not going to change it all but I think they're getting to the point where the change should be minimal you can use it today through using Conda or pip or pip end and hmm and eventually the classic notebook will be retired but if you're using the classic notebook now and you haven't tried Jupiter lab don't worry it's not gonna go away really fast it'll be several years and sort of like you know as one trails off the other will pick up I'm gonna do a whirlwind tour of Jupiter lab and I was gonna do it live but I'm not gonna do it live because of the length of time that we have and I want to make sure I get more the content out and then let you try it on your own and explore a little deeper so here you can see the launcher which is a more visual launcher you can launch different languages different kernels whether you want a console whether you want just a strict text editor to do markdown you can see that it's a similar user interface to the existing notebook one of the nice things that people have asked about with the classic notebook was the ability to clap cells and you can now do that you can collapse input/output and you can drag and drop which makes it a lot easier and in fact you can drag and drop between cells between notebooks so if you have multiple notebooks open which is really a nice feature for those of you that are using it to either teach or in production for machine learning stuff one of the other things that with the notebooks people find is okay you're making this nice notebook but I want to do like some scratch calculations but I don't want to mess up my nice pretty notebook you can do a couple things one is you could create a copy of the notebook and work in the copy so the other thing you can do is you can create a console by shift-clicking and and pulling you know just clicking like create console for this notebook and then that will give you an immutable initial content but then it will let you run things and do scratch work without it populating back to the notebook original notebook until you want it to and this one is one of those it's a little easier to see in action editors there's different editors available them sublime I think Emacs so for those of you that want a dark palette light palette there's lots of ways to customize your jupiter lab environment again connecting to the console this is an example of a markdown file that you might find as you're writing documentation and in that markdown file there are actually some code blocks and in that code block let's say we were in python we might be able to run like for small little code block well here we can render the markdown and see it side-by-side but then we can also hover over the code block I don't know if this is working but so this code block here you can hover over it shift enter shift click on it and it will actually display the output of the code that's in the document which is very nice when you're writing books writing papers things like that for those people like myself who really love the classic notebook and how clean and approachable it was one of the things that drew me to the ipython notebook years ago is I was teaching students in a Fab Lab in San Diego how to make like open hardware and code things and the notebook was the first thing that I had seen really touch a wide group of kids and by that I mean not just the kids that are interested in code but the kids that had been told yeah you're not good at math and science you can't code wrong but they felt like okay I can make music with this and that was one of the first libraries I played with was a music 21 library and since music is universal I was able to have them create music play music render sheet music convert it to Braille all in a lot of times in five lines of code or less so all of a sudden they have this new empowering experience and then with the rich Python ecosystem I was like okay pick your favorite topic and let's go find a library and you work and you be in control and I think the single note book in single user mode was nice for those people that were a little scared or tentative at first but then once they got started they're like this is eat this this is something I can do if I had known I could code I would have done it ages ago so I think it's very empowering now when we talked about it being extensible last year at Syfy I am NOT a biology person at all but some of the folks that were there created something called a FASTA viewer where you were able to I think it's genetic sequencing correct me if I'm wrong and they were able to code it up in a few lines of code one evening and make it into a notebook extension which you could then render and execute within a notebook so that ability to prototype and then change that prototype into something that is useful and will persist I think is pretty cool also datasets grids scale those of you that are working with large datasets production datasets genomic data you will probably have more data than can fit into an Excel spreadsheet and you can load that data into Jupiter lab scroll it and it moves really cleanly really nicely and it's something that with the classic notebook we just couldn't get that processing performance out of it but now we can with the more modern JavaScript so I said I was gonna do a live demo I kind of lied a little bit because I want to tell you we might still have time but I want to tell you a little bit more about how this interacts with Jupiter hub and some of the hosted tools okay so just to recap drag-and-drop functionality ability to hide cells run code blocks interactively whether it's Python law tech are Julia the link to a code console and interactively have more than one thing talking to a language kernel is really powerful and it's something that we just fundamentally can't do with any ease with the classic notebook so if that seems appealing to you that's a good reason to start using Jupiter lab and then many many different file formats a lot of different visualization formats look beautiful when rendered within Jupiter lab the slide deck I will put online and through both the conference as well as I always have all of my talks on speaker deck and these links will be live some great places if you're into science or in two notebooks in general sigh pi PI data conferences all of those talks are on YouTube so very accessible the tutorials are excellent they walk you through really from complete beginner to ok I'm proficient in both beginning topics as well as advanced topics Jason grout and some of the rest of the Jupiter lab team did a great talk at SCI PI about Jupiter lab the next generation then went much more in depth than I just did now and you know try Jupiter org should be your friend that lets you just click on a link launch a binder container and and have your own Jupiter lab there without installing anything on your own computer Jupiter hub is what we like to say is computational thinking for groups and Jupiter has been around a while and you can think of it as like these xkcd folks each of them get their own jupiter server and interface without having to install anything locally other than having a web browser we recognize that when you're deploying a production service whether it's on bare metal or to the cloud it can be complicated and so what we did last year is we sort of wrote up a zero to Jupiter hub guide using kubernetes now kubernetes is still kind of a moving target in many ways but we kind of took several different popular cloud providers gave you ok how do you get to even a kubernetes cluster on those services and then from there how do you install Jupiter hub and how do you configure it to meet your particular needs so I highly recommend that if you want to try out Jupiter hub we call it the zero to Jupiter hub guide we are also working on another deployment of Jupiter hub right now it's called the little as Jupiter hub I don't know if name will change but basically for smaller groups of folks like maybe five 250 people whereas Jupiter hub we've seen scale to thousands and thousands of people very popular machine learning you provide all the tools that and libraries that you want your data scientists to use and they've got it all there at their fingertips when they log in to their own particular user session there's also another project called cube flow so those of you that might be familiar with the tensor flow world cube flow uses Jupiter hub underneath uses the notebooks as part of the user interface as well as the tensor flow dashboard and is supposed to be a standard way to deploy different machine learning environments we'll see how that goes because there is still sort of that split between tensor flow and everything else so like Travis said binder is really exciting for us for a number of reasons one is we can share things with students much more easily we can share things with peers more easily we can share tutorials reproducible science and publishing and papers we are looking at taking binder and making it so that within a scientific paper there will be a little badge or a link that says click launch on binder and so you you might see an image in a paper and it'll have the URL that you just click and it'll load up the datasets and everything you need to run something feature nature and some other publications as well for those of you that are interested in addition to Jupiter hub you can also create your own binder hub and again we've got very straightforward instructions on how to do that this is the Jupiter team we're not a huge team so we really do rely on all of the community to help us move forward with these tools these tools would not exist if it weren't for the greater community and when we started Jupiter hub it was min Reagan Kelly and myself that's it and it wasn't until last year when we added a few more people and also got a grant do binder so we really do rely on people and also the community brings us different resources that we don't have in our own team and that is hugely valuable so I encourage you to get involved Jupiter prides itself on having friendly getter channels this coming week maybe a little less friendly than most because many of us will be at Jupiter con and probably a little stressed about talks but somebody will eventually get back to you unlike some projects we're okay if you ask some support questions within issues as well we we recommend that you try to get her channel first or the mailing list because more eyes will probably see it and then you know go forth so I want to thank you for being here I think we have a little time for questions yeah we've got about five minutes okay cool and while you ask questions I'm gonna see if I can get Jupiter lab this is like what's running behind the scenes of binder hello hi how you doing good so um can you mix different languages in a jupiter notebook and also could you explain how you got the name Jupiter okay so I'm gonna take the second part of the question first the Jupiter name came at Syfy in 2014 because we came out of the scientific community Python R and Julia are the three big languages that are typically used and it was a combination of those three languages but there's many other languages that Jupiter supports you can have multiple languages in a notebook there is like min has a project called multi kernel and so you can do it I would say you probably want a pretty compelling reason to do it before doing it you know it wouldn't be the norm but it is possible cool and I said I was gonna find cheaper to lab while you find questions hi hi girl um thank you for the talk um quick question as I was I was hearing everything kind of just like a revelation just came is this super lab kind of like a competitive competitor for like our studio and trying to get God that like share the market of like those type of users that are comfortable with like these I don't know if I would use the word competitor I would use maybe it complements it well obviously there are some things you can do with our and our studio that is really cool as most of us have science backgrounds we're not as much as I love Python and my heart is with Python and with the Python community I use other languages and other tools and you just use the tool that works best for the case our studio is supported within binder and Jupiter hub so you can have both I mean you have to have the license for our studio if it's a licensed version but I think Jupiter lab was in response to the users that we had and you know you're gonna have one size is not gonna fit all so like the little black dress that you're going from informal to very formal you know I might want to use a very simple notebook if I'm working with sixth-grade students but if I'm in a data scientist who knows my tools knows what's going on knows what I want to do Jupiter lab may be a much better approach or something that can supplement the other tools that scientists use in data scientist good question yeah so I've been using Jupiter notebooks for a while and they are great I had one comment that does the notebook support interactive commands for shell so if I do like pep install or uninstall something and it asks me a prompt do I need to uninstall yes no I can I give the user input back as far as I know yes because I done it but with bitin I wasn't sure with shell you know if you do the % % bash you should be Travis you might know but I can I think you can and you know there with the composable components things are changing very quickly as well but I am I know that I've PIP installed stuff so I'm assuming I said yes at some point but honestly I have to top my head I don't know so try it so if we use Jupiter notebooks for like generating reports is there a dynamic way to like modify them yes produce multiple yes actually I was talking to somebody earlier because we are community driven project there are other community members in this case interact and ter a CT they are Netflix is a big funder it's one of its it's open source but there's a project called paper mill that lets you run parameterize notebooks and lets you kind of take a workflow and change the parameters and spit it out on the other side and that will continue to evolve and I believe at Cal Poly this summer we have a bunch of summer interns and I believe that a Jupiter lab plugin similar to that is in the works as well oh hey thanks Carol you're welcome for the former like collaborative editing feature as a plug-in to Jupiter lab on the github pull request they were talking about looking at the atom Telegraph implementation right for collaborative editing and they used to have that for a while while Google's supported it um but is anybody looking at the like telegraph or for the collaborative so that you could teach somebody online like you right so real-time editing collaborative editing in ROS is the person that is predominantly working on it he would be the best person to give you the most up-to-date information it is a key priority it was unfortunate that that API that he had built the Google plugin and went away and so we had to take that plug in and deprecated it but yeah I think we're out of time yes but I'm sure Carol's happy to take questions I am happy to take questions all way tracks I will be here probably for another three hours and then I jump on a plane to go home and then to New York so but thank you all I hope you'll try Jupiter remember tried Jupiter Awards [Applause] you

Channel: SF Python

Views: 14,965

Rating: 4.8947368 out of 5

Keywords: jupyter, python, ipython notebook, jupyter notebook, jupyterlab, jupyterhub

Id: AXCo39qMn1E

Length: 30min 2sec (1802 seconds)

Published: Thu Sep 20 2018