Make Jupyter/IPython Notebook even more magical with cell magic extensions!

hi everyone as Alex said the my name is Nicholas show some of my excitement about bitter the artist formerly known as ipython notebook and how to make it even more magical than it already is with it's a sort of sell magic extension system so just a little bit about me I was a sort of first employee at Montreal based startup called data Craddock we were founded ten years ago where a venture-backed company and we're democratizing machine learning with our machine learning database which I encourage you guys to check out if you're intrigued at MLD bji and jupiter is a sort of ship part and parcel with ml DB it's sort of the main interface that we ship for our database and sort of how I learned to to love it for data analysis and it's basically pervaded sort of the way I approach Python in general so just so I get a level set you guys know a little bit about me who here uses the Python repple on a daily basis at work ok and who here uses ipython on daily basis org and who here uses Jupiter slash ipython notebook okay awesome and so who here comes from comes to Python from sort of like the web web development side of the world unlike the scientific Python side of the world some other part of the world awesome okay so hopefully there's something in this talk for everybody so just a little bit about about Jupiter so so everyone kind of knows a bunch of the talks that were given here where either the slides were either generated from Jupiter or just like I'm about to do the presenter basically just showed a jupiter notebook so basically in the beginning there was the python repple people could sort of type Python code executed get their answers back in real time that's neat it lacks a bunch of stuff some some people got together and built the ipython project which is still sort of console based had auto-completion much more sort of friendly console based environment and then the next level of development after that was the ipython notebook environment which was just a full-on web interface to Python and it became so popular that people actually wanted to plug different languages into it so there wasn't just Python you had the ipython notebook with like an hour back-end or Julia back-end and so the ipython notebook project has been renamed Jupiter so now ipython is sort of just the the Python back-end for it and the front end is Jupiter so that's why I'm gonna be referring referring to it as Jupiter I have to wean myself off calling it ipython notebook so very brief introduction to Jupiter you basically just launch it with a command line you know Jupiter notebook and it turns on your browser for you and and what you get here is essentially a typical sort of notebook style interface with with cells so each of the little blocks that you're going to see on the screen behind me here is a cell and so you know first things first obviously you can run you can run Python code and Jupiter so I got this little code cell here print hello world and I hit shift enter and it executes that so that's cool and then the thing that takes a little bit of getting used to is that you know objects and functions stick around from cell to cell so I can define a function here doesn't do anything and then I call it here and there we go and so I can I can modify my function up here and I can click back down here and it actually keeps again so this is not particularly fancy and inspecting objects is pretty easy too you know I type per import requests and then plate I type request question mark and I get this little sort of frame that shows up at the bottom so I can sort of look at the doc strings and stuff so that's you know it's a nice kind of web-based upgrade to what most people would be used to in a shop you can see that I've actually typed a bit of actual text here you've got some links so some cells are markdown cells you can double the plur you can double click on the cell here and type some markdown and hit shift enter and it actually renders and you can sort of switch back and forth there's a keyboard shortcut you can select the cell hitting Y makes it a code cell hitting M makes it a markdown cell and shift-enter renders it again so that's like ipython notebook in two seconds so now I'm going to show you what this talk is about which is cell magics so not all not all cells are created equal any line in a Python notebook or in a do Paterno book and that starts with a percent or a double percent is not a normal sort of python python cell they're magic that's what they're that's what they're called I'm not making it up so I'm going to show you a really simple one just sort of get warmed up it's a bit of a meta magic it's called LS magic and so when I execute it it's going to show you all of the available magic that's built in to ipython notebook and so the whole bunch of them obviously not going to go through all of them but this is sort of what's available to you and I encourage you to read through it if you just type percent magic you'll get the little documentation pane that will show up and I'll tell you all about all of the different things you can do so the the sort of one that everybody uses first because it's really easy is it is time so if you go percent time and then print do you have time it'll actually execute the Python code print do you have the time and then you get a little couple lines at the bottom will tell you how long it took to do that took about thirty-six microseconds which is good and so this is basically a good example let's call this a line magic percent time everything for the rest of the line was essentially handed over to the time magic which basically chose to interpret it as Python executed the code and then printed out some some extra information so you think you can think about it a little bit like a Python decorator the next cell here is a cell magic and you can you can tell that because it starts with a double % so a double % will essentially eat the rest of the cell and treat it with whatever magic properties you have so if I if I execute % % time it with some little loop here and think about it for a second and what it does is it runs this cell a million times and then gives you sort of a little bit of performance information so if you're just running little snippets of code you want to see you know does this run faster than that you'll often see this and Stack Overflow questions and answers people will compare essentially using this so those are examples of how you can kind of spice up your your Python code just a little bit with with a bit of magic but the next the next magic I'm going to show you % % bash it's completely different here % % bash is the cell magic that says essentially the rest of the text from this cell will be executed in a bash shell so here I just did a curl on localhost 5,000 I'm running the HTT bin http bin python project which basically is like an echo for for HTTP clients so I actually used curl and it ran and I got my you know my Sierra was down here and it dumped the output here so this is you know put this is pretty cool from my ipython notebook I don't have to just run Python I can run bash and you can you can install extensions too for running R and all sorts of other languages which I'm not going to get into so already ipython notebook is starting to feel kind of cool right you don't have to like pop in and out of the shell to do certain things that might be a little more complicated from Python so that's basically the first big idea for this talk is that Jupiter cells are not just for Python if they can tell contain cell magic you can execute code from a whole bunch of other environments including bash that's kind of a first neat thing to know about so turns out you don't have to actually just use the built-in cell magics you can download magic from the internet and install it and so for example I've installed an extension called ipython SQL so I'm going to use the magic to load some magic the load X magic to load SQL so it's already loaded because I practiced this talk but this is how you would load it if you wanted to do it from scratch and so now we have access to the sort of percent SQL and this basically turns the rest of the cell into an SQL client so to the extent that you would sort of pop back and forth between like a terminal running you know P gsrp SQL or some sort of command some sort of client for your your SQL library you can now intersperse sort of interacting with your database and executing Python code inside the same inside the same environment and the major difference here is you don't have to you know deal but with quotes triple quotes and stuff you can sort of type raw SQL so as a quick demo I the the first time I execute it it's going to take a sequel alchemy compatible connection string so here I'm going to connect to SQLite colon slash slash which is going to be an empty in memory sequel I'd database so if I execute that I'm now connected and I'm going to you know load up this very simple data set about how many Stanley Cups each hockey team is one on the top five so I'm gonna execute that and you know now I have I've essentially inserted this data into my little database and I can run and I can run select statement so here I did you know select star from hockey order by Stanley Cups one limit three and it gives me this little table now you know bunch people have used the Jupiter in talks so far no one has kind of been amazed when this one this little table shows up but I was a little bit surprised because every year exactly every example I've shown above in this notebook so far has had text right this is you know you do print something and it comes out as text and it's nice sort of it's courier it looks like code but this is a this is a table it's an HTML table that came out so you kind of wonder you know well how did this how did this the Python code I'll put an HTML table and so the answer is that when Jupiter tries to display an object first it checks if that object has a special method underscore wrapper underscore HTML underscore and if that if that method is present it calls that method and interprets the output as HTML let me just give you like a two line example of that if I define a new a new class called thing and I and it has a method wrapper HTML which outputs you know some HTML in triple quotes I instantiate an object of that class and then I leave it as the last thing in the cell which means I'd like ipython notebook to start Jupiter to display it there you go so this is essentially how you can you can hook into the smart display system of Jupiter 2 to 10 page tml into it so this is a sort of second big idea behind this talk and the thing that gets me kind of excited about Jupiter because it means you can you can hack it in all sorts all sorts of neat ways is that you can use this special method to hack sort of the the notebook output so there's one very obvious way in which I Python sorry in which Jupiter allows you to do this is matplotlib there's a special map plot little magic which when I execute it matplotlib in line basically says hey from now on matplotlib anytime you're about to display something why don't you just give me an HTML tag with with a bunch of data in as a data URI in the image source and just so it'll just show up in my in my browser so I'm going to show that off by running a little statement here select start from hockey or country equals Canada I'm actually also going to show you that for a line magic so for anything that starts with a single % you can capture the output and assign it to a variable so here I'm going to assign the output of this to to a variable called data and I'm going to call a special method on called PI and I'm going to get this little pie chart which seems to indicate that winter L which is where I'm from now is one a lot more Stanley Cups than the Toronto Maple Leafs which is shocking but turns out to be the case um it almost looks like a little pack bends eating eating the Toronto Maple Leafs so it turns out that you know once you've executed matplotlib in line this works for any any code which talks in that plot limb so this is not a very pretty graph it's kind of the default pie chart that gets outputted by the SQL magic systems but if I import any other library like Seaborn which is a really beautiful has some beautiful defaults for format plot lib and I and I get it to sort of do the typical sort of show-off I'm going to show the iris dataset you can get some nice stuff right here in the browser and so if you've ever done any interactive data analysis this is really cool right like you can have some data you can plot it and then you can go back up and be like oh you know what if I wanted to do this they can type in your hit shift enter again and you get your new to your new image shows up so you'll have to mess around with having a shell and having to remember like when you SSH into a server you got to use like the dash X option so it comes up in an X windows window that you got to close and stuff it's just sort of all in line for you right here so this is why essentially most of the people doing data analysis in Python nowadays are doing it in ipython notebook partly because you can get stuff displayed either in a tabular form or in a graphical form sort of in line with what you're doing so this is already pretty exciting and all that's happening here essentially is matplotlib is giving you an HTML tag and Jupiter is rendering it that's the only sort of magic under the hood here so I'm going to do a small small plug here although I am a Python developer I also work in JavaScript I'm the author of a library called pivot table j/s and the associated pivot table j/s Python module so I'm going to show you how you can actually do interactive JavaScript data visualization in a Jupiter notebook just by hacking so if I load up if I load up andis and I'm going to read a CSV file this is my sort of standard demo CSV which I haven't had a chance to update its Canadian MPs from the last election so I got to update this because we had so much better results this year so if I ask you this cell here I'm just going to load up the CSV I'm going to show you the top sort of five rows you've got the name of the NP their party their province their age and their gender and so I'm going to show you how you make a pivot table normally in pandas you can have to type out this whole function here and it'll give you you know like an HTML table this is all fine and good if you want to change it you want to sort of like pivot it you got to go edit this code well with a sort of JavaScript interactive system you know if you want to just load up a pivot UI a couple lines of code the the pivot UI function here will return essentially a whole bunch of HTML which will get rendered your browser and it looks like this so now this is an interactive pivot table you can sort of drag and drop and build whatever kind of visualization you want of your of your data set in line so if you want to make a bar chart let me got your bar chart looks a bit better with a wider screen yeah take my word for it but so essentially just by hacking this one little thing which means I can output sort of arbitrary HTML into Jupiter I've built myself an interactive data analysis tool right here in line with my Python code this is pretty cool right this is kind of starting to feel a little bit more magical so has anybody here ever worked with graphviz or the dot graphic the dot graph visualization language okay so normally the way you work with that is you sort of like type out a file you save the file then you go to command line and you have to type dot and remember all the options and it creates a file for you and then you have to open it okay well with with the the magic system the special magic system called hierarchy magic which I'm going to load it's already loaded I can now create special cells percent percent dot and the rest of this cell will be interpreted as a program in the dot language or essentially as a as a graph and it's going to get rendered in line right here so oh I got my graph right here so you know I basically implicitly said there's a node called hello which has a directed edge to a node called world and and so on and so on and I can add you know sort of world on fire shift-enter and now I've got my new graph so you know if you want to start like quickly making graphs this is pretty cool this is already way faster than editing a text file hitting save running in command opening it up right like that the iteration loop is way way faster and it's in the same tool that you're using to run Python code connect to your SQL database make some nice graphs okay so starting to my excitement starting to get a little you know so finally this is not strictly cell magics but it's a cool feature of ipython notebook that I want to show you interactive form so if I load up the special decorated function called interact from the I pi widgets module I can essentially decorate any function so let me show you how this works here I've got this function called echo and it's got an argument input whose default value is hello world and argument times default value is the sort of funky array and all it does is it it prints out the output or it returns the output a number of times the way Jupiter has interpreted this is it's actually built for me a little form just by essentially you're adding this decorator on to this function what it does is it just reacts Acutes this function every time I change the form so I can actually sort of like change the number of times interactively this is getting outputted hello world hello world hello world as I type here hello Nick and Friends that's pretty cool right you can build you can build a form right in your ipython notebook and it turns out that you can use this in conjunction with the matplotlib stuff I showed you a second ago to do this here so here all I'm doing is I'm doing sort of like a hypothetical version of my pie chart here where like the maple leaves have not completely sorry the Canadians have not completely dominated the maple leafs and Stanley Cup wins so you can run sort of you know what would happen what would the graph look like if the leaves had one more more more Stanley Cups than they montreal canadiens so what's actually happening here is every time that the slider moves an increment a new PNG is being generated and new PNG is being sort of overwritten directly in my browser this is not like there's not interactive JavaScript this is like machine-gun PNG generation but it's really cool when you're doing when you're doing some kind of simulation or you're doing some kind of data analysis and you want to see what would it look like if that parameter was a little bit different you know forget sort of edit the code shift enter edit the code shift and you can sort of like tweak it directly in line and we use this all the time to do data analysis sort of tweak various variables and stuff so this is really really powerful stuff all right here in the same sort of tool chain as just execute the Python code so that gives you a taste of just some of the cool things you can do with Jupiter but the premise of this talk is essentially that I was going to show you how to make your own magic extension just to show you that not only is it really cool but it's actually really easy to do so I work basically what we ship is a web service and we wanted to see if we could use Jupiter to build essentially an interface for for our web service so what I really wanted was essentially I want an HTTP client right here in Jupiter and there isn't a sort of publicly available cell magic for HTTP and maybe someday I'll generalize what I'm about to show you in sort of open source it but I'm going to show you how to make essentially your own extension right here I'm not going to live coded but almost so first things first I'm going to switch out of the actual notebook view and to do built-in text editor oh yeah Jupiter has a built-in text editor so you don't actually really need to like you know run a separate sort of program just for that it's all in your browser so here's essentially the contents of my directory where I'm running this from it's a new text file BAM I got my text editor so I'm going to call this sample X dot pi save that I'm going to make sure that I'm in sublime sorry sublime text key mode you've got them if you like so here I've just created a file command s works I can go back here what I did was actually I'm not gonna you know have to watch me type a whole bunch of code I sort of grabbed a bunch of code that I can paste to build up this little extension so here's essentially and I picked on you know syntax highlighting yeah I picked up essentially what I'm going to do here I'm just going to define an extension I'm gonna load from the from the browser so pretty simple I just got implement a method load ipython extension and call this special function register magic so what I'm going to do is I'm going to say that for a magic called HTTP I'm gonna register this dispatcher function which is right up here today all it's going to do is it's going to print a line so save that back my notebook I'm going to load my extension it's already loaded actually do this time we need to hit reload and so now if I sort of type you know percent HTTP blah blah well all it does is print blah blah so good job keep just reimplemented hello world again but now let's let's let's build this up so what we really want here is we want to be able to do sort of HTTP you know get and then something here so I'm still running my localhost on my sort of echo server so what I want to do is to be able to actually type you know HTTP commands here have them execute and see the output right here in my in my output cell so in order to do that all I really need to do is fill in this little dispatcher method here right so I've conveniently written all of that already so you'll have to watch me do it so I'm gonna fill in my dispatcher function I mean essentially all it's going to do is it's going to very sort of very very quickly part of the line because the line is the input of this one of this the cell magic and if it's again I'm going to do a get using the request library if it's a delete I'm going to delete and I'm gonna Parrs essentially the remainder of the after the URI as some sort of input so save that go back to my magic and this time I gotta be loaded obviously reload this time tada it says response 200 okay so I've actually you know I've executed some sort of HTTP request and is giving a rich response 200 why am I not seeing anything cool here well because in my extension all I'm doing here when I'm when I'm executing get as I'm returning the output of requests right which is a response object and the response object doesn't have my fancy rep or HTML method so it just is just using the default you know underscore underscore wrapper which in the case of a requests response object is just the return code so what I'm going to do is I'm going to use this little thing that I've already written which is essentially a decorator on top of my dispatcher method which allows me to decorate it and sort of pretty print the output so that's what I'm gonna do I'm gonna decorate the response to the dispatcher and what all it does is if it happens to be a valid requests response object it's going to add this reparation ml method which Jupiter will call sort of pretty print so load X temple txt and now tada I've got sort of you know color coded output it's showing me my my my return code and so the HTTP bin server that I'm running has all sorts of funny options so if I if I do this status 400 it's gonna show me 400 bad request if I say 418 its gonna say HTTP 418 I am a teapot you know and you can actually you can put right if I put two slash put and I'm going to give it some JSON so a B then this you know this is executed to put and what this echo server is telling me is that the JSON that you put well I'm going to give it back to you to store a test that it works so that's all you really need to do to build a Jupiter cell magic I've just essentially extended Jupiter for my own purposes so you can imagine that any kind of service that you are interacting with that you want something that's more terse than actual you know full-on Python code with all the the quotes and everything that you're going to need you can build yourself a cell magic extension for people have kind of gone a little bit wild with this stuff building all sorts of extensions like I've shown you for SQL and various graphic backends I'm surprised there isn't one for get yet or at least not one that I found but it seems like an obvious kind of extension HTTP would be nice to have a sort of fully featured HTTP client but that's basically that's basically the cell magic system so just just to recap I've shown you that you know you execute Python code you can write markdown cells you can also execute bash code you can execute sequel commands you can exceed HTTP commands you can draw graphs you can draw the other kind of graph like directed graphs you can make interactive data exploration tools available right in the notebook you can actually code Python you can write Python files right from right from Jupiter so I hope I've got you guys a little bit excited about about this tool I've got sort of one more thing any notebook like this one can be served up as a slide deck really really quite easily so the first thing you need to do here is kind of you toggle the the the toolbar so when you got the toolbar active in in Jupiter you can see that there's this little cell toolbar here if you turn this to slide show you can essentially control for any given slide how it will be outputted when you put it to a output into a slideshow so I've already annotated this whole notebook so you'll have to watch me do it but basically I can just grab this loop command line command line command switch over into my sorry it's already here in a shell I run this so I said you could run bash commands in ipython notebook you can but long-running processes like this one just it's not very good like it's good for interactive type work but essentially this exactly this exact notebook that you saw me sort of scrolling through and executing code dynamically is now a nice little slideshow nicely formatted you can scroll it's got sort of everything it's got the output and it's even got the graphs right here in line so you can take this and it's one HTML file you can email it to someone you can stick it on github it's pretty neat so some of the some of the speakers of this conference already have used this system and it's great it's really awesome you me a little progress bar and if you guys can see the top few pixels here are like a progress bar for my my presentation and speaking then this is sort of normal reveal.js type magic and speaking of hosting this stuff this this notebook is available on github and if you actually go to this link here in a new window you will see that though github well actually any Jupiter and Jupiter notebook that's sitting in github github will render for you in the github web UI so you can actually just like scroll through this thing and read it without actually having to download Jupiter and run it yourself so that's basically the end of my talk I'm happy to take any questions and I'm easily reachable in any of the ways that are available here on the screen if you have any questions please line up here at the mic yep so you say so um so I pile on notebook sorry Jupiter is essentially a giant security hole just to be clear not only can you execute arbitrary Python code but it actually comes with a web-based a web-based terminal which is really convenient when your new terminal so you can literally have like a shell here so there's there's nothing built into Jupiter or other than a basic password system as far as I know the way we do it is that we only bind to a local port and we require you to SSH tunnel into the into the server so this in terms of like the way we use it we've just punted on that and said it doesn't bind to a local port or to have a public port and in general it's something that needs to be you know you need to be aware of when you're running a local web server that is allows you to run arbitrary code just sort of be aware of what you're doing yeah I'm sorry need could you repeat the question oh so we can record it right the original question was uh what about security basically over there yeah you know you do away who I warned up code part of or users reporting there is no built-in feature for that and and you should email the the organizers so that the oh sorry the question was is there any way to sort of hide or control lock the code output for business users so there's no there's no feature in Jupiter for that and you can email the the developers to ask them why that is there are a bunch of different ways to do that I have like a special JavaScript snippet that will essentially hide the code when you output when your exported to HTML but there's no built-in way to do that so when you do the slides essentially you can't just show the output of a graph you have to show the code that that generates it with this all like new to you guys or is this all like old hat than you were bored by my presentation you okay old hat and you everything okay okay oh I'll be around for the net for the rest of the conference you guys won't ask me any questions or one of more demos or something like that do you know of any extension or magic that does interface with Bev Scheel with WebGL there are products that essentially exploit WebGL in their output I believe graphics tree is one which does sort of high-performance graph visualization but there's no sort of general-purpose WebGL so magic all the ones I've shown you it's barely fairly straightforward to write one that answers your question q hello time for one more question anyone okay I guess that's all the time we have for this talk thank you very much

Channel: PyCon Canada

Views: 47,517

Keywords: IPython, Python (Programming Language), Jupyter

Id: zxkdO07L29Q

Length: 28min 51sec (1731 seconds)

Published: Mon Nov 30 2015