IPython and Jupyter in Depth: High productivity, interactive Python - PyCon 2017

all right it's nine o'clock welcome everyone we're going to do a tutorial about introduction to ipython and Jupiter and and and how these tools can help you get your your daily work done there are three of us going to be presenting different sections there's myself I'm min or come on been working on the ipython Jupiter projects for a long time and then we have two other instructors Mattias and Mike over there so you could raise your hand and people could see you in that can help out you can just if you have an issue you can just raise your hand in front of them or one of us that's not presenting welcome help you out so to get started first you want to get the tutorial materials which you can do you can get the tutorial materials from github we are in fact going to do the tutorial presentation with Jupiter notebook so we'll be doing learning about the that will by using it and to get started you can download the repository with Rickett or you can visit the the website and and download an archive with from the github website with the download as a zip button whatever you're more comfortable with and then to get the software as in the email that hopefully you got yesterday you can install ipython and Jupiter with either pip or Conda whatever your your preference if you're using Conda you can just Conda install ipython and Jupiter and you can download Conda from the continuum website the quickest way to get a really full-featured Python environment for analytics and and science and computing is to download the full anaconda distribution so if you just google anaconda download you can get started with with python with everything in that comes with Jupiter and I installed or if you prefer to use tip 4 for pure Python things you can install ipython interpreter with pip and before you install as I would like to encourage the practice and there will be a talk later by matias in' and others during the conference this weekend but i encourage any time you're going to install something the first thing you do is to upgrade the pip itself to make sure using the latest version of the installer because it helps with transitions to Python 3 into klika so once you have ipython Jupiter and you have the folder called ipython in depth from the repository in any terminal you'll want to change into the directory called ipython in depth that you just created and run the command Jupiter notebook and once you've done that you should see something that looks very much like this so everybody's yet I'll set with them and just raise your hand if you have any difficulty getting getting notebook running in and opening the browser that is and well well everybody get started on that home I'll just give a bit of background on a night Python and Jupiter so there's just to clarify Jupiter came grew out of the ipython project so ipython has been a project that's been around for a bit over 15 years now focused around just a better experience for interactive Python and it started with just a terminal interface and with better tab completion and introspection but it's grown and we've added new features and new new interfaces some new ways to interact with Python and one of the things we did was this notebook that we're looking at right now and when we built the notebook one of the results was that the notebook was actually useful in languages other than Python and so people started using ipython notebooks in Julia and our and and Scala and other languages and so we decided to Jupiter part of the project or the notebook part of the project that wasn't Python specific needed a new name and so then I Python notebook got pulled out of ipython and became a standalone project called called Jupiter enesta that's the relationship of ipython interpreter that Jupiter is kind of a new name for the parts of ipython that have that can be used outside outside Python and so the first part of this tutorial work we're going to talk a little bit about this notebook application first and then and then we'll go into ipython specifically in how ipython improves here the interactive experience of working with working with Python so if everybody's all set with their notebook dashboard the notebook I'm I'm starting on right now is this index notebook so you can click that and you'll get a table of contents showing well that rough outline of what we're going to cover today with links to the dental books that we're going to be that we're going to be using and we're going to start out with this notebook basics where we'll see a bit about the notebook application and how it works so if you click that link you should see you should see a notebook title - notebook basics and we've all gotten started with launching the notebook by typing the tubular notebook command at the at the command line and this takes a bunch of options that you can use to control how the notebook runs but by default when you run this command it opens automatically opens a browser pointed to the notebook web server and the notebook so the notebook application is itself a web application so all the UI runs just in your browser and on the notebook is a web server serving files and running code so the different pieces of the notebook application the first piece you'll see when you start the notebook is the dashboard which has the list of files that you can work with and can have different files you can have notebooks but not just notebooks you can also work with text files and and others and if you go to if you look at that dashboard you can see that you can check boxes and when you check boxes on on on items you can see different actions available to perform actions on those files so I can say select this file and I can make make a copy of it I think duplicate and yes I want to duplicate of that now I've got a copy I can rename and then you can also click Edit to open files in a a text editor so normally you've got this and you can browse into folders by clicking folders and they'll just open a new and a tree view and you cannot appear with the breadcrumbs you can navigate up to previous directories so we're working with this notebook basics notebook and another tab in that dashboard next to the file list is the the running tab and that tells you what is currently running so when you open a notebook it starts what we call a kernel which is a process where your code runs when you actually type code enough to execute it and each each notebook that you open gets a different kernel so when you run code in one notebook that's running in a different process when you run then code in another notebook so if you you know create figures define variables those are all available in that that one notebook and then when you use a different one then you've got a different name space different process where where everything is running and because these notebooks can be in all kinds of different directories there's this convenient list of currently running currently running notebooks so I can see here index is it's green because it's running you can see there's this little running label and you can see when it was last modified in this case in about an hour ago and you can sort this list by last modified date if you want to see you know what are the most recently edited files in this directory and things and the next part where we'll focus most of our attention is on the notebook itself so when you click a notebook from the dashboard you'll get a view that looks like this where you have don't hide it you'll have a document style view of the notebook document itself and then you've got a familiar kind of file edit view menu bar that you can click and perform different actions and a useful thing here is user interface tour that give you you can open this to take a little tour through the notebook interface and we can see that's in the toolbar and in the header here we can see the file name this is the name of the current notebook we're working on and if we click that name then we can change you can change the name of the notebook and you can see some lasts whether there's unsaved changes or whether it's been when it's been changed and then the main part of the notebook is this document area where there are where notebook is a sequence of what we call cells so different each segment of the notebook is a a cell of one of a few kinds so this notebook is composed just of markdown cells where each of these cells is actually contains markdown which is then rendered to HTML but when you're running code then there will be code cells where you can type code and see and see their output and there's another cell type that's for more advanced usage is called a raw cell that's used for a tool that will be talked about later called env convert where you can type some lot tech or some some other format and then that code will get passed through to an export file but we won't get into things that that is answer today so when you're working on a notebook the thin Epoque has what's called a modal interface and so there's two modes when you're working with the notebook when you have a blue border on the cell you're in what's called you're in what's called command mode where if you start typing what that's going to do is actually issue keyboard shortcut commands so there's no there's no modifier so if I wanted to cut a cell I would just press X instead of command X if I wanted to copy itself I just press C instead of command C and there are shortcuts for navigation for moving cells around and then to enter edit mode when you want to run some code or edit a markdown cell when the cell is selected like this one we can press ENTER and that enters edit mode and now now we have a green border and we can actually edit the contents of the cell in markdown so I can text them and then we can leave edit mode by pressing escape and then you can route you can rerender the cell by pressing shift enter shift enter does is it takes a cell and if the code cell it executes sit and if it's a markdown cell it renders it and selects the next one so you can select cells by clicking on them scrolling around you can select multiple cells by holding shift and clicking adjacent cells and that that lets you select a group of cells and move it around in the document so for instance I can select this cell and this cell and I can move it up and down in a document with these arrow buttons here and in command mode you can navigate through the document with the arrow keys just select the next cell by pressing down preview cells up and you can execute if I had a code cell so I did create a new cell above and this is this is now a code cell you can press the little play button to execute that cell so if I click the code one then I can press this little play button and it will execute that and you can see the result is is one Android there are a bunch of keyboard shortcuts that you can use in this command mode and the way you can see those keyboard shortcuts is in the help menu you can click keyboard shortcuts and that will show you all the set of shortcuts for in edit mode and command mode so you can change the cell type to code to markdown you can cut copy paste save etc and then there are some keyboard shortcuts in edit mode with modifier keys so some common shortcuts that are there there are lots of keyboard shortcuts and you can even customize the keyboard shortcuts to it in a notebook 5 you can customize the shortcuts to suit your preferences but common keyboard shortcuts that four really common operations that are good to know are a and B for creating a new cell above and below so I can select this guy and I can hit a to create a new cell above and I can it be to create a new cell below the current cell and keep creating cells these are useful for quickly creating new cells and then you've got cut copy paste etc and the standard keyboard standard shortcuts just without modifier so x4 cut C for copy V for paste and then we have keyboard shortcuts for interrupting the and interrupting execution with I I for interrupt and period for actually restarting the curl now the notebook web application is not just for editing notebook files there's also the text editor that we saw so if we open a we go to the tree we can say click this new button and you'll see you can create a notebook and each notebook is associated with a different kernel and so you'll see a list here of all the kernels you have installed on your computer and you can install kernels for python for julia for our for scala for for SPARC for c++ for any of a number of languages and when you click one of these it creates a new notebook but you can also create things that aren't notebooks you can create new folders or you can create a text file and if you create a new text file that will open in a full-page plain text editor so this is I've got untitled text but maybe I want it to be you know test dot pi not a Python file and I can right something right some Python code and the last piece enough yes yeah so the question is about what's the workflow of saving and and kind of houses it is it talking to the server and working the files so it's it's really just like working this plain file so when you save it's just writing a file to the disk so doing things like working in a get repository and writing commits and everything is handled as a separate activity so it's just like opening a file in your traditional text editor or IDE and then saving that to disk and then doing a version control operations separately so it's not it's not doing anything special in that regard it's just creating files and modifying files on disk and when you're running well it happens to be a web application with a server the way we're running it today is it really is you can think of it as a death local desktop application where the interface happens to be in your web browser so you're just editing files on your in your home directory on your local disk so there are hooks the question is about are there hooks for doing more actions on on save and things like that there are hooks in something on the server called the contents manager that implements this saving and opening of files and you can register additional actions to be called on save which could could do things like create new commits in a repository add file to a repository that kind of thing so there there are hooks to enable that kind of thing but they're not there aren't any hooks enabled by default and so the last piece of the notebook web application which isn't particularly useful for the local case because people have a terminal but since because the Jupiter notebook is a web application it can be used to expose remote computing resources so maybe it's running on a hosted server as part of you know company infrastructure or campus computing resources for a course and maybe your only window onto the machine and if that's the case then you can start terminals on the notebook and this does require that the server is running on a UNIX machine a Linux or Mac OS if this isn't this one feature doesn't work on on Windows but if you create say a new terminal then what you get is a familiar terminal interface and I can do things like LS and get status and so you can do if you're doing work with notebooks in a git repository you can use this was weak would be where you enter your your good commands for creating your commits and things and this is a full terminal so you can run things like VI Emacs and all those fancy things yes so the interface of the terminal is running in the web browser the there is an actual real PTY running on on the server so the these are actually the files on my file system but the UI is you can see this just is my safari browser so this is just an another web page exposing that terminal so this lets you have have a terminal in the remote notebook case this lets you have a terminal access accessing a machine without having to do things like SSH and and deal with that kind of stuff and back to our index we're going to get get on to running some code so we're going to open the beyond plain Python notebook we're going to get into so I was kind of a bit of an overview of the duper a notebook application and how it works and what what pieces there are to it and now we're going to get into using the notebook with ipython to actually run run some code so the first thing is we select our cell or first cell with a bit of code in it and then we can press shift enter to execute that cell and select the next one yeah this one if you go back to the index you click ipython beyond playing python and I'm going to clear all the output from this notebook just so we can see the results as we're as we're making them and so you can print prints and text a lot of I pythons extensions to Python have to do with introspection and getting help and figuring out what you want to do so a lot of the time when we're doing interactive computing we don't quite know what we want to do and you know maybe you know that I want to use this library function but I don't know quite how to call it I don't know what it exactly what it does and a lot of ipython is built around asking questions of of your data asking questions of ipython itself and most of these things are implemented around the question mark which is something that I python adds beyond what the Python language gives so the most basic question you can ask is just what I tell me something I don't know and so if you just type question mark you'll get some output that tells you about ipython itself and the how to get help where to get more information what can I fight on do you know what's I fight on for and then a quick tour of the kind of information that we're going to go over today so kind of when in doubt question mark yes yes there's a thing called the the pager and there's some configuration for whether output that would go to the pager should be produced as normal output which is what I have here or it might show up as a pop up from the from the bottom and that that pop up is called the pager and it lets you kind of see a health output and still scroll around in the notebook but that's just a display preference so that will give you some help about kind of how I python how to use ipython the various things I press on can do and and where to find more information the first thing that will do most often with with ipython is say I have an object in this in this case the named tuple type from the collections module and I want to say tell me something about that object and so if you type a symbol in in your namespace and you put question mark after that and then you execute you'll get the signature so now you know how to how to call name tuple and then you also see the doc string for any documentation that the object has about itself and this really helps for you know you're using some some library numpy matplotlib pandas and you know you know you want to use a you want to instantiate one of these objects or you want to call this function but you're not sure what the argument order is or not you're not sure what flags are available to to steer that to what options are available to you question mark can give you a really quick reference to answer those those questions that may not be without having to you know in separate documentation and deal with switching back and forth to different contexts so in this case I can see in creating tuple you give the type and then you give it some fields and then there some keyword only flags and then some there are some examples for how to use it and now if you want more information so that was question mark if you want more information you can give it an extra question mark just say you know tell me more so if we do that again instead of showing just the doc string we actually see the source code for the file for the for the object so I wanted to look at the collections counter class and if I give it two question marks I actually get the class definition so I can see there's the doc string there because that is attached to the class but I can also see actually how the init method is defined I can see the counter is a subclass of dictionary and then I can see what methods are defined on it such as you know the most common item the most common items in the count and the counter should give me the elements of the counter you can this can be useful for you know inspecting maybe you've got a function and it's doc string doesn't provide enough information to tell you what it does you it's you think you you understand what's going on but it doesn't do what you expect you can just peek at the source code of an object or a method and see exactly what's going on and it can be useful for the kind of poking around exploration and ipython and then the next use of question mark is with these wild cards so here I've got asterisks on either side of int int and then a question mark and what this does is it searches the current namespace for any symbols that match that that pattern so in this case it tells me that I've got the built-in function print int and I've got the floating point error and now this can be useful for maybe you know a module that has a really big API and you want to find where a given find all the functions associated with a something or you know you kinda remember the name but you don't remember exactly what it is so an example I can import numpy which I know an umpire does a lot of things with arrays but maybe I don't remember exactly what method I want to call so I can say show me everything in numpy that has array in the name so I can see I've got an umpire as array array equals wrecker a and so this lets you let you do things that are more than just have completion which only match the prefix that lets you say show me everything that mentions or everything that starts with array or ends with array that kind of thing and then like the question mark we've got a quick reference so just a little helper card to show you the kind of common actions you can do as i paysanne to get a quick reminder for when you're getting started and and trying to get used to what ipython can do for you the quick reference will show you okay I remember question mark I remembered the wild cards and then the magics that we'll get to in a bit tab-completion is an important part of any interactive interface so when you start start typing some text and then you press tab if there's only one match it will automatically complete that text and there's attribute completion so I'm working with the collections module and I can press tab and it'll show me all the attributes on that collections module and I can just use the arrow keys to select a match and I can type once I've opened the tab I can start typing a sync generator and this this is useful both as kind of poking around tool so we mentioned the the numpy array so I just lets me do poking around like that wildcard match but this lets me do prefix kind of as you type poking around and it also lets you can see what a attributes an object has so I've created a variable I and I can do item I can see all the methods on that object like I can get the conjugate I can get nominator imaginary part real part when you're doing interactive execution so each cell each one of these cells is a what we call an input it's a code that you've that you've asked to run and each input has an output so you can see on this left where it says in 11 that's input number 11 in that counter goes up as you execute more and more code and that's not necessarily related to the position in the document although if you run them from start to end they'll all be in order but because this is an interactive session I can go up and down and run things out of order so I can set up for 11 and I can run another cell and so output 11 is below output 12 and if I run this again those will get different numbers so each time I run this number goes up the previous output this is most useful in when you're using ipython in the terminal when you just type ipython in your shell you can get the previous output is referenced by the underscore variables every time there's a there's an output we can reference it by the previous previous results as underscore and add 10 to it since I just run the cell this cell is now the most recent output and refer run it again it will add 10 to that so we've got 12 and then 22 and I can keep running that and it will keep increasing by 10 because it's getting the output from the previous execution and I specifically the previous execution nots the previous am in time not the previous execution on the page so when you're running code that's not tied to what it looks like in the document that's tied to actually the time order when you're actually selecting cells and executing yep so the question is about interest and intersecting with the terminal and the relationship with environment variables so I will skip back peek ahead a tiny bit and so the the notebook server itself is just a process that started on your machine so it has that it has environment variables defined when it started a kernel that kernel was a sub process that I that may have gotten environment variables added to it and it will also inherit environment variables from the notebook server environment and so if you you have so in Python you can access environment variables from the OS module so if I look at dict OS environ we can see these are these are the environment variables of the process and i python has what's called a magic we're going to type % ends and that gives you the same thing and it'll work so it's not that common so there's four Python things to be influenced by environment variables but those that are they'll be they'll be available and defined and and and have the appropriate behavior in I - just like they do in it in a normal shell both in the terminal and in the notebook some calls have expressions have an output that you don't actually want to see so maybe you've got a cell that you're you end the cell by doing a function call maybe that plots and that that function call returns something that you're not actually interested in you can suppress output by putting a little semicolon on the end of the line so here I've got 10 plus 20 which would normally output 30 but if I put a little Senna colon on there and it actually won't show me the output but it does still store the output it just it only doesn't display it so the semicolon suppresses the display of the output but not the storing of the output so if I look at what the or it does suppress the story on the output so if I look at the previous output that's this environment dick there and then you seen the little out guy over on the side those are available as Python variables so out the capital o is a Python variable that's a dictionary with integer keys so I can see what's out 10 is and there's also a shorthand variable with underscore maybe 10 this is what I get for executing out of order I'll go with 11 and you can see what out eleven was was twelve that's this guy up here and we also we also have so the outputs are cached with this out variable and the inputs are also cashed with this in so each each coat each bit of code that you asked to execute is available in a dictionary called called in actually in might be lists where you can access the blocks of code that you've run before and you can modify them you can see the most recent input with just underscore eyes like what was the last thing I executed and then we have these magics so magics R or something where we extend the the Python language with these percent percent character so if you start a line with percent that's going to call the rest of the line in this case this - n one two five and it will show the eye Python command history for items 1 2 5 and the magics are another thing that we're that eye Python provides that extends the paevey of the the Python language and gives you new convenient functionality for kind of common operations that are possible to do in Python but often might be tedious like CD you can just type just like you would in the shell you can type CD to a directory instead of having to type out like Oh SCH dura thinner so as a quick exercise for knowing that percent history will tell it show you the ipython history and knowing that we have this question mark tool for poking around and learning about about objects and functions we exercise for putting those two to work is if you can figure out how to with the history magic right the last ten lines of your history to a file called log pie hmm underscore I is the most recent input and so this is the most recent input you said underscore eleven that is output number eleven so underscore eleven is shorthand for out eleven and underscore eyes is for input and you can see there's actually I 11 so I 11 and 11 you can see are the input and the output number 11 and all of the kind of terse features like these underscores are from ipython terminal interface we're saving characters is really precious because you're typing everything out and you all the time a lot of these things the more terse synonyms for something we don't use so much in the notebook because typing out characters is not as not as costly in the notebook yes ah yeah so the question is how do we manage overloading special variables in most cases we don't just you can just like in Python in general you can use those variables to define to define things yourselves for the most part we don't we will define those variables but we don't interact with them and so you can override those functions just like you can override the print function and that will generally be fine it's just that the ipython variable won't be available to you because you replaced it there are a couple where I python interacts with those objects like out where I think if you override out to something ipython will complain the as it tries to put new items into it but for the most part it'll just be it will just be fine you defining new variables so the first hint on this exercise is a question mark it's the key to finding out new information so if I do history question mark I can see the doc string for history and this is an exercise in reading back strings so my task my task is I want to write the last ten lines and I want to write it to a file called blog that PI so what I'm interested in is okay how do I tell it to get ten lines instead of five and how do I tell it where how to save that to a file instead of outputting it to my terminal well we can see we've got - F will say instead of printing the output to the screen send it to a file okay that's part one and then - L with a limit says get the last end lines from all sessions that the all sessions thing is because I python saves your history from each time you use ipython it creates a new session in your history and so ipython stores your history of commands for not quite forever but for a very long time up to 10,000 commands or something so you can see you can search through your history of commands for the past couple years and say I know I did some complicated plotting call I don't remember exactly how I did that let me search through my command history to see figure out - try to remember what the exact right incantation is for this complicated library so based on this information I know I wanted history limit last 10 lines so limit that to 10 then I want to save it in the file log top I want to overwrite yes they can get you can see that not especially interesting commands because I was demonstrating via input-output cache English recently only this session yes so if you do by default when you just type history it will give you your whole this is my whole session so that that's actually the default behavior is show me this entire session yeah I know so it's it's ten so it's ten lines going back from the end of this session to the beginning of this session and then to the previous session so yeah yeah so it's not going to do the last ten of every session it's last ten commands and it will cross sessions if you ask for more commands than you've executed this session and then there's a look there's a little query syntax for sync specifying which session and which lines in that session if you want to say you know show me you know to ipython sessions ago the first five commands that kind of thing if yes so this notebook is not next to the index notebook so another feature of the so we've got these notebooks distributed throughout folders and then notebooks in this tutorial are in these subfolders by category so there there are a lot of resources in this tutorial that we won't cover today because it's it's enough material for three to five days of suit of teaching so encourage you to check out the repository and poke around with the stuff the other the more advanced things so we don't cover today but when when you open a notebook the kernel the process where that code actually runs will start in the same directory as that notebook file and so since this notebook file lets perfect lead into the next thing that we're running this if you start a line with exclamation point what that will do is execute a shell command so in this case I'm executing the shell command PWD for print the current working directory and we can see that we're inside the repository in the example subfolder and in the ipython kernel subfolder in there and so if we yes so this is just calling out to your shell this is it's really just calling out to OS system so if you're on Windows you can do bang Durer and all the windows shell commands and if you're on if you're on Mac OS or or Linux and you've got all your all your UNIX shell command PWD happens to be a command that is the same on both I believe and if we do you can store the output of these shell commands so here I did files equals bang LS which is say run this shell command LS and store the output in the Python variable files and then I can print you know the current files in the current directory and then I've got a Python variable and what is that it's a list it's a Python list where each item in that list it's okay Matias is it's approximately a Python list you can treat it as a Python list it's more than a Python list it's a Python list with some fancy convenient features and the each item in that in that list is a is a string that is a line in the output of the command so if I did say LS here you can see each each item each item in this listing and you can see each item in that listing is one of the entries in that files variable so that lets me get the output of shell commands into Python but we also have the other direction you can also get Python variables into your shell commands this gets you kind of a fancy you know shell scripting environment with with access to to a lot of Python features so I can do exclamation point echo which is just the echo command but then I can give a dollar sign just like you would do for an environment variable in a shell but I'm going to give it files which is a Python variable not a OS environment variable it's just a Python variable in my current namespace and what that does is it replaces the string you know dollar sign files with the string representation of that variable in my current namespace and you can also use Python string formatting so if you put something in braces then in those braces you can put any expression and that expression will be evaluated in the ipython namespace and then that's the string that will be assembled onto the command line that gets executed so in this case I want to say give me the first item in the files list and then make it uppercase and then pass that to echo writes and I'm echoing the personal first file in uppercase and these kinds of operations don't have to be on whole files of their own whole lines of their own they can be or they don't have to be in all cells of their own they can be kind of in arbitrary nesting structures so I'm doing I'm iterating through the files I'm saying if that file is a notebook then echo a you know the index in the list and then the base file name without the file extension so this is mixing Python formatting shell escaping shell execution and you know Python iteration and logic on all together and then if it's not a notebook just give me a just a little doodoo and there you can see in order we get the titles without the file extension and then we get just little placeholders for all the files that our notebooks so the last and biggest extension that ipython makes to the Python language is this notion of magic so we saw a little preview of magics with history a magic is anything that starts with this % in ipython and just like we did with question mark the first magic that we look at is the magic called magic and what magic does is it tells you about magics so if you do % magic it tells you about what magics are how they work some common magics and in fact a huge amount of information about every magic currently available in that buy in that Python ultimately every magic is just a Python function but instead of that python function taking you know being called with python objects it's a Python function that will get the first kind of magic is called the line magic so it starts with 1 % and then followed by the name of the magic and this one called timeit and then what this gets is the rest of the line as its one and only argument and then that time it function isn't is responsible for interpreting that line as something and then taking some action with it and so the time-it magic which is particularly useful in ipython when you're doing profiling when you're doing a little quick performance analysis of what it does is you give it an expression or statement on the right hand side and it runs that and then tells you how long it took and this is this functionality provided by the Python standard module standard library in the module called time it but using the Python module time it requires a bunch of explicit arguments and you have to tell it how many times to sample and how you know how often to run and the time at magic just adds some heuristics to do that to pick a sensible number of loops based on how long it takes so that you can you can do this really common task with the with a bit less typing so what time it does is it takes the expression it runs at once figures out how roughly how long it takes to run and then it does some statistical sampling of okay that that's a really fast operation I'm going to run it a hundred thousand times and then do some some metrics on on that and if you do something if you time something that takes a lot longer it might it might only run it once or ten times to because it doesn't want to do a timing that'll take three days so in this case it's creating this this list of a thousand elements takes about 15 microseconds give or take give or take one microsecond and so that's line magic switch get the left the rest of the line as their argument then there's cell magic so we started with one percent for a line magic four cell magics we start with two percents and just like just like the line magic the cell magic gets the rest of the line as an argument if there's anything there and it gets the rest of the cell just as a string to interpret however at once and this lets you do things like in this case time you know give them do the same timing for the entire contents of the cell but it also lets you define in this case the rest of the cell is Python code like a normal Python cell but it could be any there can be any arbitrary interpretation of that string so there are for instance SQL magics where you write write just a cell full of SQL and it executes those queries against against a database and there's that there are magics for R and Julia that actually executes R R or julia code and send data back and forth from your Python interpreter and can do arbitrarily fancy things with these guys blind magics can be used in code blocks just like we saw with the exhalation point for shallow escapes so you've got a for loop and I what I want to do is create ranges of various sizes and I'm just going to iterate through through a number of a number of sizes and then time how long it takes to create those those lists and then it will pick the number of loops based on how long each one takes which are very similar but you can see that once I crossed a threshold of two microseconds it switched from a million a million loops 200,000 loops to keep the the total size that time and within a reasonable reasonable range and so because cell magics can have this arbitrary interpretation one useful form one useful application of these cell magics is calling out to other shells so there are a lot of scripting languages you have on your computer there's a good chance you have bash Perl Ruby R etc and you can I Python find several of these shells and gives you registers a shell magic for each of those in this case bash where the code I'm writing here is not Python code this is this is code interpreted by the bash shell and then when I put % % bash that's as a I python instead of executing this code yourself take this code and actually send it to a bash interpreter and execute it there and then capture and display the output another example of a cell magic which takes both a line and the cell is the right file magic and what this does is it takes on the line you pass it a filename so I want to create the file test txt and then the rest of the cell is just the content of that file all right so I this is useful in tutorials sometimes when I want to say okay I've got this tiny file on disk rather than opening up another tab and saying look at this file I can just say I'm actually going to create the file right now so what this does is actually created the test that txt file and then I can look at the contents of that file and then to finish it up for future reference there's there are lots of of MAGIX that come with like Python there are also lots of extensions to ipython that you register new magic so ultimately a magic is just a Python function and you can there are extensions to ipython that say you know add this new magic so the SQL magic is not part of ipython it's a separate package that extends ipython and gives you these new magic so siphon compiler has magics for doing in line compilation of saipan extension code which is really handy for for profiling and prototyping extension code and there are lots of these fancy magics for getting a new custom functionality and tie Python and you can write magics yourself or for common operations that you happen to do a lot that you want to make more convenient and you can register those with your Python installation by default so LS magic just gives us a quick listing of all the current magics registered in ipython so I can see I've got this HTML and SVG and history and and all these magics that that we haven't used but I bet you can explore on your own time yeah automatic is on by default that means I can do things like what did I just do time it time yes you don't have to and this is a preference that you can turn on/off so you may or may not need to so I can do the auto magic magic toggles whether magics are automatically executed so far now that I've turned on auto magic and I do that I'll get anywhere that the auto magic isn't defined so if I turn it back on and then I can execute it it will turn it off and that won't work that's just to save a character kind of thing that sometimes documentation will have code in a form that looks like the Python interactive Python or ipython session that has these prompts on the front and so this is something where often you'll have documentation that has years into some example code and this is what it looks like in the terminal so it's got actually the prompts and so if you just select the big block and copy it it will include prompts like this so this is not valid Python code but ipython knows that this happens all the time that people you know grab some snippet of code from documentation I just want to copy that and paste it so I thought some recognizes these prompts and actually strips them before executing the code so you can this ipython recognizes these prompts and says ok strip those off and I know where the Python code is in this block of text and then execute that and the same goes for what a typical ipython terminal prompt which what it looks like and then I believe the last bit here is dealing with exceptions so python has a lot of information about errors that occur and when you're doing interactive computing errors will occur a lot because you're typing and you know you make mistakes and do things wrong and you're learning and figuring things out and so good error messages are really important and and dealing with errors and poking around and figuring out what happened is a big part of the process of doing interactive computing and so here we're just going to make a file that defines these two functions and then I'm going to call them I'm going to import that module and I'm going to call the function in such a way that it raises an exception and so this is what exceptions look like by default in ipython there's a bit more information than Python shows you by default but I can also change use a magic called X mode which is changes the how ipython prints errors by default we can say plain so that's too much information X mode plain calls that down a bit to a more sparse a bit of information to be a bit more compact or you can say you know I there's a lot of information here I'm really trying to figure out what's going on you can enable for both exception mode and what this does is when you see the error in addition to just showing the stack it's actually showing all the variables and their current values at each stack frame so if you're having trouble tracking you know why did this raise why was why is this one something over zero what's the value of y and everything the verbose mode actually shows you at X at every stack frame it tells you in this frame it tells you I got the variables F and Y it tells me what F is and what Y is so Y is 0 and s is this function and then in F you can see that x equals 1 so it actually shows you the value of the locals at each stack frame which might help you kind of in the debugging process of figuring out why you know maybe you it makes sense that there was an error with that value but like why was it that value I want to you know I want to poke around and figure figure that out the default x-mode is this is this kind of inter intermediate mode that we call context and we can run additional other shells like Perl and Ruby so we've got this divided by zero error and I want to go debugging when you use you ask for input in the notebook you'll actually get a little a little prompt to to actually input that some response to to any prompt so if we use the debug magic so I just raised an error and now I want to do some post-mortem debugging saying okay now let me give get an interactive session that let me poke around those stack frames like a new percent debug and you can do debugger commands like BT for back-trace go down the stack up the stack and then say okay what's why there what's F and then we can just ask for input the Python built-in input function or in that the input function lets you ask the user for for input and then the return is the string that says that the user typed and then the last little bit is plotting so MATLAB is a common plotting library in in Python and we can do some simple plotting and this gets produced so we've been dealing with text output up until now but I notebooks can actually have output in a variety of formats in HTML Javascript PNG etc and so when you use MATLAB you get these outputs in an image format and you get those inline in there and then a little bit of bonus so do the way execution works in in the notebook is that the kernel so the notebook the browser's talking to the notebook server over the web but the notebook server is actually talking to the kernel over over an additional protocol and the way that protocol works you can actually have multiple front ends connected to the same kernel at the same time and so maybe you have this notebook but you want to switch to a more terminal like interface there are a variety of interfaces for Jupiter kernels there's a terminal interface there's a QT console interface that's kind of in between the notebook and and a terminal and the connect info magic tells me of what the current the connection info that I need to connect another front end to this kernel so there's the connection information dictionary that tells me all the ports I'm using and and keys and things and then there's some handy command line commands that I can execute for how to connect another another front end to this kernel and then there's a magic that actually does basically just does exactly that of it this just calls Jupiter TT console - that's existing and then the kernel connection file and what that does is it opens a tiny tiny tiny tiny tiny intact opens a QT console and this is connected to the same process so this is just another UI connected to the same process that I can switch back and forth and so I can say I've got fig is my figure and I can see that in this QT console and there are some some behaviors that can be a bit more convenient to do in a console type interface and I can I can type and execute a little bit faster and a little bit more efficiently for debugging and poking around in the GP console than I can in or the terminals and I can in the notebook and so this lets you switch from the notebook where you're producing this nice document over to a more terminal interface where you're you're doing a little bit more poking around with in kind of a scratch scratch area and that is the gist of the first segment and we're supposed to have a break at 10:15 is now 1010 which i think is close enough so if people want to want to break for 10 10 minutes or so any questions after the first segment yeah visible so the question is is there any way to have a tickle TK interface that's visible in the browser the answer is no so you can't have TK creates things that show up in the browser but you can if you're using the local notebook like I am right now where the server is actually running on my laptop in addition to the UI I can run the tickle TK and it will create Windows and those windows will open and I can interact with them if you want so if you want something that for remote a solution for remote TK anything that creates real Windows via a native actual windowing system your only answers are kind of actual remote desktop tools so there are you know no machine virtual desktop B&C all these things or tunneling X Windows so that that kind of solution is your only real option for libraries that talk directly to windowing systems that said technically you could implement you you could implement something that works in the browser but it would it would essentially have to be a TK replacement that instead of talking to the windowing system talks to something that generates HTML yes yes so each cell or the question is about updating seeing the output from one front end and another one the console outputs do support that it's not on by default but there is a flag when you start the QT console to say if another front-end produces output show it to me there's there's just a flag when you start up those those additional front ends for whether whether it should win it because those they do receive that output and then they make a decision about whether and how to display it the notebook doesn't display it because the way the notebook is essentially defined is that each each one of these executions is an explicit like create a cell put in some code execute it and see the result so we don't want all the executions in the in another environment to add a bunch of cells to this document that's part of the reason that we actually switched to that other environment in this case was we wanted to do some poking around without creating cells that we're not interested in it's the information is available and it's technically possible it would be possible to say when I run code in another front-end to create a cell here that information is available in we could do it we just choose not to essentially there was yeah so the the out this is something in Python called the display hook so when so cells can have output so this is a print statement that wrote to standard out but if I just do one so this if the last should be careful about terminology I'm going to get it wrong if the last statement in the cell is an expression with a value that's not none then I python will automatically display that and store it as the result of the cell so you only see this out if the cell has a result so since print returns none there's no result so we don't save it and you don't get an out prompt but since one has a value we save it and store it yes a terminal not a cutey console so it's not regular ipython but it is in a terminal what was it yeah and then there's this so this is Jupiter app and - existing and you don't really need yeah yeah yeah it's connected to this thing that is just if I yeah it shouldn't be one it just hasn't asked what the number should be yet so I can do fig which opens up in a window in this case now it's actually 10:15 officially break should we should we restart is everybody very ready they're critical part of that machine but just I'm going to make it announce before before we start we have a repeater con in August in New York City so if you like Jupiter and you want to attend awesome dogs you can go to Jupiter condo org it will really relaxed you there and we have stickers and cards if you want to to get also the information for for later to click on your on your laptop and everything and feel free to spread the word that we looking ahead and even we have awesome speakers it would be a the first time we get a grip zircon Fenton in soup rods great what we try to get one every year okay so let's let's continue for the second part of this tutorial okay so meaning and did this presentation at the perfect time let me talk slowly to get you avin back into your computer what we saw there I was mostly features of of the Python canal and Jupiter has a really difficult distinction between the front ends and the camels and you saw just at the end that you can have many fun things for the same the same cameras and photons may have some slightly different different features so for example when you're using the ipython channel you might get different things depending on whether you're using it from the notebook and undetermined so I'm just going to show here two small difference between the front ends so here I have a notebook and I'm going to import numpy and if I want to do to get some information for example on the Empire that histogram so I can complete and if you open an a bracket and press shift tab it will give you directly in line some information about about histogram and how to use it if you are writing a really big function and you don't want to do an empire histogram and then question mark to get the information at the bottom it can be quite useful and this tooltip is something which is specific to the front-end to the notebook front-end which you don't have for example in the in the ipython terminal and though you have things in the ipython terminal that you don't have in in the front-end at least not yet so for example if you create a new terminal in to zoom in a bit and start ipython and you will see that since one of the latest version now you have code highlighting in the terminal as you type so you can define Fibonacci function that takes input N and if n is lowers and chews and return 1 else oops we turn fib on s minus 1 atom plus C of n minus 2 so first what we will see here that in the terminal we also have multi-line input so we can go up and down which most rappers don't don't give you that's because we're using something called prompt circuits and and what you can do is add new line in the middle with control which if you don't you don't do that but the fact that we're using prompt circuits allow us to do some realized completion and so for example if I import numpy and I do numpy that histogram you see that now at the bottom of the computer it will tell me the type if it can't enter it but it will tell me the type of the object is completing to that's something which is not yet in journal book and which can be useful and that's currently on the stable version of ipython and just to give you a small sneak peek because you're here on the latest version of a five Python if you do if you try to complete it will eventually use a signature of your function while you're competing so you have features like that that are core to the gamma medal of the language you're using so the magic bank to execute shell command is specific to the Python canal you have other features like tooltip and some UI concretion that are specific to the front end and that's sometime confusing when you're changing languages and that's what you are going to and to see now is how to use many languages with with Jupiter so if we go back to Python in depth I have seen data already running and you can find in example ipython kernel that's ipython support which outputs so let's let's go there and so um one of the features that's not book front-end support is for objects to display themselves not only as strings like in the terminal but as images as math and sounds or any other mime types that that you want so if you're writing a library and you want your object to be displayed in different way you can define specific methods on these objects that allow you to display images so ipython provide a function which is the equivalent to prints what we call display and display will allow objects to show better representation richer presentations and just strings and it's situated in a Python display so usually when you want to start voting with ipython you want to import that provide Python itself we don't inject it by default in ipython because if you want to let book and you export it as a Python module we want everything to be - we don't want people to be confused like this play wouldn't be defined if you run a Python script miss pure Python so here we are explicitly kissing how we are in ipython during putting that it wouldn't work if you are in a pure Python wrapper and we have other convenience methods that allow you to filter the representation and object gives so even an object is able to show itself as PNG as latech as SVG as JSON you want to be able to say show me only the specific presentation and these are also in ipython display and so for example if you want to to show an image we provide a see also a convenience class called image and so for example what I can do is I can load an image assign it to a variable and now when I return the value when I return subject when the subject is the last statement of expression of a cell it will be assigned to the output number five and there is a slight difference between returning the last object of a cell and displaying this object so here I display hi and it is that both are almost identical except here I have out number five so if I try to accept access out number five like we should use so prover previously I would get this image I would forget to switch objects while out six will be would be none and this is the same difference in the Python ripple when you print the variable on when the variable is the last value of cell and you have a values and values option you can show things from your health so if the Wi-Fi conference were was down I would have an image not found here and you can also display SVG so here I have I have an ECG so depending on on the kind of representation you have if you want something that is scalable and you simply might want to use SG we have a couple of of convenient things like when you have an image do you embed the data of the image into the notebook or do you just stop link to the image by saying this is the height of the image so for example here I'm not going to execute this was the Lawrence Berkeley Hall of Science last time I ran this notebook when I run the cell above here the data was embedded into the notebook and in the next cell where I said something equal true and this is actually currently pulling from from the live feed so if I were to refresh the notebook in ten minutes this image would change and this image would not of course if you want to target reproducibility you might want to to embed your your things you can do the same with me exactly more compact with a lot of different convenience methods so for example if you wanted to display HTML in that book because you want something to be well represented you can use the HTML object which is again in ipython display and now you create the object and you can say this page and if you've been using dumbbells to select a frame that what combat is using and so your data frame will be represented as nice HTML and even when you hover with your mouse on your row of peppers it will be highlighted so you can easily see what's happening if you're just trying to throw some HTML on the page one thing you can want to do is use the HTML magic so here we see the percent percent HTML it's available by default in the namespace and you can really consider the cell magically doesn't take in your arguments on the first line and you can write HTML and again you will have a change here you have a lot of convenience method here you can also display JavaScript and here I'm going to execute this that will display oops that will display some JavaScript that will run on the page and so now we have my page I'd say hi um there are some some security concerned about executing JavaScript on the page and you will see here on the top of my screen it says that like that book is not trusted which means that I was not the person who produced this notebook which means that outputs that are insecure won't be shown by default it's one of the things where when you're sharing another book with someone you can get a difference you might see something on your page and they might not because they might not have executed in that book so you have two ways of trusting in a book and getting all the rich outputs that can be for example dynamic graph that you can score that you can manipulate and do brush Direction either you click on not trust it and say well yes I trust the person who made this done books is not book is not going to destroy my computer since the book is not going to steal my credential and so forth and you click trust and that book will be trusted and reloaded on the page and then untrusted output will be shown because he that put is everything which is HTML Javascript and may execute code on the page or the other possibility is to go to sell all outputs clear so it will remove all outputs and then you want to run all cells if you run yourself all sides of a notebook you're executing all the code and then the will be journal book will be automatically trusted as being yours if you're working on multiple machines what may happen is that it will be trusted only on some machines there are way to synchronize that but but that details so keep in mind that if you share network with someone and you have a difference in appearance and elude you load one book it might be a question of trust and look at this this this symbol we're just going to to skim through the different things you can do you can come you can display some some complex JavaScript and keep in mind that if you display JavaScript or HTML it might not work on all front-end so here is something that will be specific to the front-end that allow you to this page on a script you can have a nice math if you're doing applied math or any thing where you need equations like the community ISM you can display them as actually nice nice nice l'attaque and we haven't provide if you're writing Python 3 if you actually want to write math with with Unicode symbols you can do sita tab and it will give you a theta and then dot and i will he visit adults that I can assign to certain regions and it will just it just works that works only on Python 3 but it's quite useful when you don't want to have compact equations so you can display the tech and you can display audio if you sing recently some gravitational waves were measured by the LIGO interferometer and one of the things it did is to actually release not books at the analysis and at the end of these notebooks they produce two audio files that are actually frequency shifted version of the gravitational waves we can literally listen to the two black hole the bike riding so you can you can have video enjoyed you go find those I can directly have yo in my notebook which might be useful if you're in ecology stand off ticking birds or all different kind of of animals so that's just a quick overview of how you can you can what you can and we can display another book of course by giving videos and a lot of other stuff and you might want to define your own representation so the example I can command so if you go to example I Python canal and custom display logic we can see how to define your own objects that can have rich representation the thing to keep in mind is that when you define certain objects at for example define latech and PNG and text when you display the object everything is sent to the browser and only one is shown but when you save it all the three are saving to the document which means that if you convert after the fact to for example PDF the PDF exporter might want to extract a different representation from your object and show it differently so for example if you have I if you're working in neuroscience and you're showing a brain in in doodle book on the light network in HTML it might be a 3d model that is represented you save it between the exports to PDF so pj6 buffer might say well i am charlie actually going to to display not only not a 3d model but slices because of course on my PS and i print it i can't rotate my brain so I I want to have the multiple slices so the way to do that to let let's import again display a macro clip is to define specific methods on your objects that are underscore wrapper underscore a name which varies which tells a Python what is the kind of representation I want for my objects so for example pandas that I frame will do an HTML and I think Allah take one and when you will display this object so ipython will see that it's there call it get it out send the data to the front end and the photon will display that so for example say I'm working on probabilities and I'm creating a Gaussian objects what I can do is I use a specific method to pre-compute a graph and I also want to have for PNG and I also want to have a type of presentation which is my Gaussian is just this mean and this standard deviation because runs out to PDF I don't want the whole image so I have my gosh and object and what I can say is create a Gaussian object with mean mu is two and some navigation Sigma equals one and what you can see that it will automatically when you return it from a cell to automatically show the right the right description so the front end has a preference of what you see it thinks the best representation is but you can tweak that and say well I actually would prefer to have change a--first off-- refer 12 AVG first and of course you can also call the display on this object and it will show on on the screen without the outer number here and same object I'm not by completing anything I'm just going to ask ipython filter and keep only the PNG data and I get TNG data for for my Gaussian so you can explore the different representation of an object without having to and to recompute all that the only the back here when I'm calling this play PNG is that it will actually strip all the other representation which means that now here in the document I only have the PNG the data subject so if I convert to PDF here the F would have to embed the PNG because the math and the retic is gone and so you can you can try to change the number the number of examples and you can display display to Goshen index next to each other and if you're developing a library that there's a lot of parts to using a lot and you don't have nice representation if you're always in same thing you might want to create this representation for for your object so for example let's say you're working with numpy polynomials I personally have issue reading that you know that it's the value for each for each power of X and you can say well actually I want to represent my polynomials zwei so how to convert an umpire in a mural to do a tech representation and so you see that if you call you call it on the podium yell it out life that God is not yet represented and now I can just pass that to the right at function and now I will have my nicely represented represented polynomial and what you can tell Python after is well I don't have access to Mumbai I don't want to change the source code of Mumbai but I as a person prefer to show this object in this way and you can tell that it's well how can I tell you the way to represent objects and so here you see that I find all knows how to display some building the object either as SVG JavaScript plain text JSON etc etc and you can teach these things how to display each object and you can here get em latech formatter and say you can look at at the help which tell you exactly what are what also different as a different signature and then you can say hey i python here is how to represent mumbai polynomials and now suddenly my polynomial that was showing us at strings before has a nice the type of presentation and makes it much easier to to read and to to go through your day when doing when doing and data science if you really have some complex objects that need to do several step things like fetching JavaScript to get libraries and Sunita you can completely customize how ipython is displaying objects by using a rich silikal ipython display that we will not see right now but for example we will put a play with it but you can you can use a JavaScript like d3 to display the spider plots here it's neither SVG nor PNG it's something called slot and so this is part of the Jupiter protocol like an object can the tributo protocol send something across the wire and so if you develop a camera for Jupiter I get less of your language you can make use of of this and you can have R you can have Julia you can have Haskell to display which objects the exact implementation on the camel side will depend on on which like Julia does not like to different method on objects they define as I have a registry of how to display object and you don't define than the rate parameter like we do night Python in Haskell it's again different and now what you are going to see is how to install multiple multiple cameras for 4gb there so let's go back here it's good example not book multiple language and important so when you're working with with repeater you might want to work either with different languages or even when you're just using Python you might have to work with different environment in some environment can be in vitam 3 some other in Python 2 you might want to have several version of a library in different environments and you don't want all that to conflict you might event wants to have a statistic process that ran on different machine so for example you might collect a lot of data and you don't want to move the data around in your lab so you might want to brighten candle to actually start in your experiment room or you might want to work on a specific beam line if you're on a synchrotron and candles are basically that kind of things they are not always one language there are one way of starting a computation process and this process may be one language it may be many languages and that's how the drips are protocol really it really helps because then you can just defer to anything that implemented for the call either camel and the camel can be whatever you want it can do whatever it can be a chat about it can be anything so at the base I Python used to be is a rapper so basically you have a while loop you get codes from the users by using input and you figure out some way of executing the code and sending that back to the user that's roughly after is just cherry on top you just make things prettier and what we did is um abstract that into the network protocol which is not the Jupiter protocol and so you have the core execution um which knows how to get information from the user execute it and send something back and then you can choose how do I get this information and send this information back to the user and so terminal ipython when you're in the terminal and just type ipython and get special prompts it's basically saying I'm going to hook into my core execution engine but I'm going to plug into it via standard in and sell out and what we did is basically say well actually we don't want an audience now out we're going to switch to a network protocol and we're using this your mq 0 mq is relatively with the available in many languages it'll be easy to to build it to build on so if your language has 0 mq binding there are a couple of methods you can implement on investigate protocol and then suddenly you will get a camel for your language or or your project let's pick the trickier protocol and you can jointly make use of all or also for them you have different ways of going of important internal either you're saying I'm going to implement my channel for my language in the language itself so that's what we did for for ipython ipython is in Python but you may want to say actually I'm it's a bit complicated in our to write vmq bindings because our doesn't have good packages but I may want I know that are as really good Java bindings and so recently we got some people who said well we wrote an archive net in Java and so there was already Java bindings for 0nq and nice Java bindings are and so now there is a our Canaries in Java there is also an archival which is not native are but you can you can decide to go to go to different route and so um when the language and that you're executing is not the language used to write the caramel we call that wrapper camels and also have native skylight and so the way it works is you have users your smiling because it's a bike on and you like that and you have your browser and your browser is actually speaking through web sockets through the network to the netbook server which holds all also notebook documents and as access to your file system and your kernel is even remote from your notebook server which means that you can not only have your servers after all your document in one place but you can have even your canals that are remote so you can even more distribute execution and each Kandel may even talk to a cluster which is a different machine so each of these boxes can be on a different machine so if you have a lab and you have AV computation you can actually really really distribute distribute all that it might be a bit complicated in some case like when you're running everything on your laptop at the pass to your different files is relative to choose a notebook server and the path of the file that you see when you're executing code in the kernel is depend on where your channel is on your process is on your own file system we have the confusion down here like when I write log files to disk yrn say next to where I started mine observer when your camels can are on different machine you can see a complete and file system and something that you might want to keep in mind and so now the question is I have by driba tur with my Python 2 or Python 3 candle right now you can see which color you have by going to the CalNet change can line and see which color you have here you probably only have one and the question is how do I get more so what I'm going to do is install more languages so if you're using Conda or Python you probably have already Python installed you might say well I want to create a new kernel and register it with with Jupiter the way to do that is in Python you create a new environment you get into this environment you install the camera into this environment and then you register the canal with with Jupiter so here are the instructions I'm going to to do to do it live and create a camera named named icon so here I already have Julia our Python 2 and 3 I'm going to go there to do new terminals so I have a new terminal need to do kondeh create you can use your channel and if you don't have Conda I'm going to give it a name name like on 2017 and I want right on 3.4 we're going to create sub Python 3.4 environment you can you can do it at the same time and event try try variation if you want so that's let's say yes and so in the under Greenland I don't need to install off of all of Jupiter I just need to install the camera it's a bit slower I shouldn't have done Python for what before the right entry she could have in cash I just need to install for Python I need to install ipi canal so what you can do is source activate the environment and in general and once you install ITI canal you just need a cue like allen and all the library library you need and to give a series instruction so what I will do after I will install iti Carmel and then I'm going to ask i pol Qi Canada to install itself and give it a name so name you can give it both a name which is machine readable if you on something which is only lowercase as ki-44 machine mexico central purpose and you can also give it a user friendly name which is I don't know tompa Hawaii whatever because I'm going to try new libraries or you want to have a stable stable environment on a dev environment so maybe try this python 3 6 and so once you register your Yochanan it will drop a specific file on your file system that tells all of the Jupiter ecosystem here is how to access my camel while it's doing that I can show you how to do it with the archival as well and so if you have Conda you might you might want to install our from the arc channel it's here so - he R is saying look into the arc and I'll install our I also wants to install our IR channel and with our the way to install it slightly different you need to start an hour session and say I have canal in star speck and what it will do is actually drop a file again on the file system that will give me access to 2r and then are the archival should just appear here in your menu if you refresh the page and if you're not using ganda you might have to compile things on your own if you use install packages these things and then this tool installed github by a camel ir colonel it will do not things compiled things automatically and once this is done you can go back here and we do our kinetic instance back and it will do the same installed specification that connect back on the right drug directory if you are the command line you can also see where all of your camels are installed you can issue jupiter can inspect lists and it will tell you where your available camels are and you can see here that i have a Python to camel which is my system one which is in use or local i have Python three and Julia are actually installed Gioconda and you have this one which is in is per user so what you can have it have you system admin installer cameras that are global to all users and these are likely to them and if you want to have a specific version which is just for you because you're messing around with some libraries you can install on in your home directory and then your divisional book can switch between each of those and so now I can do source activate icon on that install I why multiplied - sometimes insulin doesn't always work because it can take quite a long time to to compile I've deployed a couple of one hour ago if you go to this IP there is a server which have you can put any username and login and it's already has our Korea Python 28 entry installed and you can play with it if you are not able to to install it on your machine so you just put whatever you want let's mash the keyboard not be not not using semicolon and put whatever you want have a password and sign in and after a minute or so you should get a server that have all the tutorial material and python 2003 Orangery available this is top row rates will stay up only for today but you can play with it in that what we're going to learn how to deploy in the third in the third section okay so it's installed at the same time and now I can do Python - am i - I can I'll install - - I gave it a name my super icon KML good permission denied - user it's installed I go back to my notebook I'm going to refresh the page and hopefully if everything is right I know have my superbike on camera so I can switch to it and it could be a different language if you have been doing it with are you should have an arc online available she did it with Trulia you can have a dragon available and now I can just either when I create a new notebook I can select my language directly create a notebook notebook position or I can actually switch the language of occurrence my book is going to kill the current canal and restart anyone with the spective you selected and so here now I know I'm not an hour person I'm sorry you can you can actually run our and it's just works as long as it implements the protocol you can assign get nice nice in nine o'clock so that's the first step and the next step we're going to see is um one of the records we have is well and that book is one which I actually want to have many canals per notebook how can I do that and it's often the wrong question because the canal is the thing that allow you to execute code a specific canal may not be a single language and one demo you have here is that you're actually using a Python canal to run those Python and julia and the way we do that with the ipython journal is to say well we are going to tell us our ipython canal when we want to switch language by using magic because the magic doesn't have to be right on syntax so what we are going to do is um load Judea dot magic which is provided by the Python julia package and what is this does is it injects a percent percentage area and percentage area in the namespace so we can use cell magic or julia and line metrics for julia the cell magic for julia allows us to execute code in a julia process and the line magic allow us to send things back and forth between to the n pi down so here is a python process and the tree opposites are persisted not awesome like other magics other magic are just starting a process running some code getting the richard button killing the process here you have to live processes that are talking to each other and doing things and so what you can do here is say hey Julia import method clip from Python and important and buying from Python oops you take sometimes because it started ready interpreter and that's a magic and there's a hoot I don't understand and then you can say Julia use what I just told you to import on Python to draw some stuff and so you see here I have link space in space is different if not something we know in Python it's not in the bed in the nominal space so it's Julia pie is also truly arm and T we just defined before it's Julia sign is Julia and P cos is vampire so we are using an umpire cosign he is Julia TLT comes from Python it's metric lib get configure and let's plot T s in red with the language with - - an indelible so that's just Julia and after a few seconds I get a nice figure and so here I use the Python process to talk to Julia and and God and God sings backs and now I'm looking a while but I don't like to interact with matplotlib via Julia because I don't know the old recent acts really well I'm going to pull the figure back from Julia into Python and modify it so you can use the present Julia to send things back and forth so you can say well Julia fig as I need to figure so this thing is Julia the rest around is Python what you can do is web plops a sign on top of it I was not really inspired by design and so now you have this cross language things you have in the same process you have a common entry are just using using logic and just to be a little bit crazier because this is just sending data back and forth we're going to define Fibonacci if you remember Fibonacci you have Fibonacci of n of n which is Sybil energy of n minus 1 plus C 1 F G of n minus 2 either you recurse implicitly or you're crazy and you're passing is a function you want to recurse with into Fibonacci I'm not going to write Fibonacci in Judea that whicker's using the Python function and the Python give energy functions that recurs using the Duryea Fibonacci function just to see if it works probably not but let's try [Music] so you can see that in this cell I define pretty tersely Fibonacci injuria that requires using bicep and I define Fibonacci in Python that recurs using fib and fib will be Julius it and I'm just going to not execute it just to total thrill so it compiled and can I run it yeah it works but I can even do is Prince P every time I move from Python to Julia and J every time I go from dry to Python to see the internet attacked rice and I have Python to the Altria Python Python to the background so you can do some pretty neat cross language things internal book if if you wish to and it seems really magical but it's not that hard and that's what we are going to to see how to do that by writing ecstatic so let's go to example no exercise sorry exercise crystallization crystal magic and cross-language integration so a quick reminder magic our escape hatch that allows you to inject whatever syntax you want your in your ipython process you have line magics at our present magic and get a single string and % % magic that get both the strings down after it and all everything which is which is abused and we're going to define a simple tick-tock magic just to get our hands dirty I give you some some things on how to start you need to import something from the kaurav I buy down to teach ipython that's some function or magic so you will need register line magic from ipython core magic and register line sell magic register line sell magic is telling ipython that a magic can be both a line magic and a sell magic and we are going to to import inspect to inspect some sub stuff we can look at documentation for or register line magic and say well use it as a decorator um without argument you can always argument also if you want and you define a function underneath and you can you can be tagged with the rest and it tells you that it will actually take the function and register it with with ipython we will also inspect compile because you might want to compile some source code and probably executed and try to see if we can make a tick-tock magic so what we were going to try and not not look at the solution but do something like register like magic and death tick-tock it's a magic so it's going to it's a line magic so it's going to take a single argument which is a string so it's the string user wants to execute you don't have to create that long and you want to get given time execute user code and get new current time do the difference and return the difference I will give you a few minutes to try to do that on your own and see if you can get a tick-tock magic if you on the solution it's a just video it's just uncomment and and run that it will give you the solution I'm going to try to do it at the same time micro sequences maybe maybe two motive too much chicken service to make the seconds but he wants to do strange I'm using ship type to see the signature and it's the source so so string it's a fine name let's icon and we want a mode we're going to go with exact and then we're going to like that that's one for its close to nothing it will be a basic magic but it should work and let's go with stir for ster turn faster - before we have now tick-tock we have kicked up here and let's do let's do one five two four microsecond we can compare that with percent time it's one plus one that is way more fancier that's what we did in five minutes well there is some overhead but you get the idea let's check my solution I need the data but your friends love doing it okay and I was named error by CB is not defined in my my version I lied coded I didn't ask levels so it couldn't get access to functions defined integrable in a place you just need to look at Python documentation to know that but you see that defining a magic for yourself if you have something which is that you already opened you and going functions are difficult you can just pretty easily do that on the fly and because your magic is now returning something what you can do is use it in in the middle of Python code and you might want to look at your Fibonacci that I define here which is a bit ugly choosing the dummy recursive algorithm and see how long it takes to to run I'm going to give me some results I give you some functions to plot it and you can see that so this is a similar plot and you see that there is a nice straight line that appears here and it's pretty slow of course I'm not doing the right algorithm but it's pretty slow and I might want to speed things up and often when you're doing some speed computing or anything else you want to have just a chunk of code which is really fast little for sometimes we might skip that um the time it you can look at basically how time it magic is using different options what what you can do is to time it and do two question mark it will show you the source and you can see how I'm time it magic is doing most of so that's doctoring most of after work together and toolbar and so you can see that the timing magic is way more complex is actually a class it has a lot of moving parts s the namespace injects things in the namespace but you don't need to go all that fancy unless you have some really complex complex things and so we want to to go faster because our Fibonacci is too slow so we're going to say well I want to embed C in in my Python and I really don't like most of the libraries that are around because as in said earlier it need a lot of boilerplate and if your bottom is for the don't like boilerplate or just wants to get work done you might want to just write a bita magic so I give you basically how to write a Fibonacci function C if you squint closely you see that there is a big mistake but let's not fix it now we have the header and the source and if you go online and you say well how can I compile C on the fly in my in my Python process and get a shadow black which is fast easy well you probably want to use the FS I see if I use nice it's not too hard it's only 29 if you want to write to write a function so let's let's import the FFI oh it's not there we're in ipython so what we can do is add a set above and we want to install cat5 for the q and python and so we are going to use a bit what we seen earlier we are going to use this executable which will be the path to the current python minus MP install 5 Vincente iff I should and now we can be important it works and you can look at the CT pie documentation I give you a give you most of the thing here you will need a file name because say that I will compare things in to find name Dutch art objects you need to create a native file context set the header of the function you want to compile sets the source of the function you want to compile with the file name then you ask csfi to compile the subjects it gives you a shadow object and then you want to import this hard-charger object because it's dynamic you want to use the under imports or import Lib I'm going toward endo import and then you want to inspect these objects and see that it has something called g-sib because if you look at the source above it was ship it's here and you want then to get the ciub object from the module that leave that stupid thing and you can look at the documentation and it's your your function so now hopefully you can you've assigned see fit to this and you want to measure it and it should be webmaster and that's just showing you how to do that that's the orange stuff and you see that it's at n equal 20 it's way better still not right because it's a dummy algorithm but you just gain two of the other of Manatee in speed but that's still a lot of boilerplate so what you can do is write a magnetic clasp with some information here can you make something which is percent percent see that take your C function body and compile the function you wrote there and inject it into the namespace that's um that's exactly have just after I give you the basic template of how to do that here we are using a class because when we instantiate the class we give the class access to self dot shell which is the current ipython session which means that now you can poke around the user name space and you can look at what are the things the user has defined and even define things for the user so for example if if you do that you can do um user NS which the user name space and assigned to the variable sip so function that does nothing I'm going to leave that you that and one thing to know is that you can't reload shadow object in Python but you might want to do is compile it with a random name import it from the name so that it is different every time you redefine you rerun the magic so that you can basically have interactive see you basically the person person see you define a function as your function is wrong you fix it you rerun it and your function object in your namespace have change so you can um we'll give you site five minutes to try to get some of that if you really don't want to do it but you can do this tipping store yes si magic and poke at the source imports CFS hi magic oops actually it's already there and you can get some some hints of how to do it okay three minutes is not a lot but we have a third section so um you just take what's above you can just go beyond basket and you see that um it should technically just just work on otherwise you can poke around us in here you can see that I have my um random name that I have my FFI contexts I set the header I set the source with my random name I compile I import my module and I will just loop around the attribute of the thing I just compile and set the username space to set the value in the username space to the function in the module name and now it's registered and I have my function and what I want to do now is look at C seed I and PI sub I for I in range 10 C well is my C correct syntax I forgot parentheses and you see that strange my C function and my Python functions should have the same value right well I forgot here it's not for it to first abuse it might have been more sneaky to do something like that where it's just a single equal but the thing is you can redefine your C function you can just do that and now you have a c4 it's a c python extension that just has been recompiling there is a hood and we import it in your global namespace and now your value are are correct and this function is two of the order of magnitude faster than the python function and so if you want to go much faster you can now just fishing using that each iterate over your problem way faster without having to go through a lot of boilerplate and you have nice syntax highlight which you don't have if you use a Python string we can compare the speed and we see that probably yeah the C function is blue now and you see that it's about to order of magnitude faster than my my Python function and just a side note that of course implementation is Adam algorithm and if you improve your algorithm you can get way faster actually it's not C is Python now time you algorithm you can see that python can be way way way faster than C if you use the right algorithm of course you can use the right algorithm in C and then B again way faster then then Python again but choose your battle it might be easier to write the more complex algorithm in Python because to be shorter is and to get it right in C so you have to to create flexibility and speed of writing for for speed of language execution and so all of that was running from from my laptop so you see that here I have I have a C function and I could also use a Judea magic which means that in a single notebook I could use Duryea see I can use an R by 2 magic which are you to have are in the same process and you can then in your single notebook with a single camel you could write something in five six different different languages there are metrics available for fight on for Fortran for rests for C C++ I don't know for Judea and all of that should if you manage to install it which is not always easy actually just works after easily and the way to the rapids would be to often use docker container and you might want to deploy that on the cloud and that would be a third speaker we take place here I don't know if we have a break again or not so I'm going to leave this is a stage and we're going to deploy things on on on Google cloud thank you and come to come to to Peter Kahn yes so you have ways of doing that you might want to install get clean and smudge filters I don't I don't have them install right now but get clean and smart filters are basically operation you do on the netbook when you commit them from the working directory to your to your index and back when they go from your index to your working directory so for example what you could say is when I commit this notebook to my repository remove all outputs and so the committed version wouldn't have any outputs so that's a simple way but for example what some people are using there what you can do is actually explode a notebook two separate tubes multiple subfolder or convert a notebook to markdown so when you commit the notebook so the notebook is actually committed as markdown and when you check out that book it goes back from Mac down to like OMB and under the hood you just don't see doubling for you it appears like it's a notebook but then for the version control system it's going to be to be marked down notebook are in general challenging for for for versioning system and mean is working on and done I'm which is different large do and be behind it which is not in the time and so dime is for disk merge and so basically it gives these extra hooks but if you have conflicts you will get with this UI which is which understand the notebook format you need to configure gates to do the right thing that's all the instructions are here and then you can choose like here is the figure before here is a secure after which one do you want and then you can resolve your merge conflict in a graphical way of course github does not understand that and you need to resolve the vertical ethically but anyway you always need to reserve the msconfig kukri you might just have more merge conflict does that answer your question - not we're also working on a real-time collaboration and if you've heard about sage mad clouds it's an something that hosts Jupiter it's not the only only thing but they have real-time collaboration they don't support all the features of objector yet but if you not working with git and you want to be several people on the scene that book at the same time that will handle with the multi cursors multi people you can a chat on the page and we are working yin is working at Berkeley he's not here today he's working on integrating Google Drive with a normal notebook front-end so you could hopefully in a year something like that you have your local install of Jupiter and you say well I want to collaborate actually right now with someone click the share button say well I mean can you help me on this notebook I mean get an invitation and then I can see me doing change online a book there are some complication like when you have real-time collaboration who owns the kernel and what is the execution model is obviously a question that we're honoring because if I'm sharing something with you and I execute something do i execute something on my computer and you exhibit on yours in which case both can L get out of sync because you can't share lies in together but in one thing you don't want is that when I press shift enter you don't want me to execute code on your computer of course and that's that's the challenging part of of collaboration also on the book is that you have not only the document has a static state but the document is often coupled with the dynamic state which is the state of the camera and that's why merging the book always not always make sense so if you take a notebook like this one if you go to this you either is this not book exactly because there is no no password right now and we both collaborate on that it's like having to open windows of Emacs and each of us pressing pressing save is it will clobber each other as they will share the same canal so if I execute a equal a plus 1 and you existed a girl a plus 1 then which would be a girl a plus 2 but that's not you should not collaborate using a single notebook server you should go a layer above that we're going to see with Jupiter hab and real time collaboration is not something which is ready yet unless you use a service which already implements implements it and each user have their own server and this should be a certificate to their own server because their server runs as then you don't want to have someone else access to your server and so that's why if you look at me look out there is a huge warning set up HTTPS and set up a password because even on your own machine you're exposing a network port and someone who is not you may send request as to your server if your server doesn't have a password initiative yes it's you technically you can not do it but that's how the current I'll pretend works so say if you spawn two servers on the same local directory and you open the same netbook they will have different canals it would be different server process that we write to the same file it's like opening to Emacs windows or one index finger than one being windows and once you bang windows you and technically is not the contents manager may not be as I stem it may be a database so you may be whatever you want but that's hypothetical Mathematica so if you're doing that that's not directly related to Jupiter because we are basically just allowing Lagrange's how to migrate from Mathematica to do something like that we don't have a nice 3/4 articles people can use whatever libraries they want underneath the hood we are just basically pretty pretty here on top if you're doing symbolic math you might want to look at this in PI but after it's a question of what are you doing with Mathematica because Mathematica tend to bundle everything in the question will be like what what is what are the libraries that are useful for Yero use case if you're doing sadistic so you might want to look at Seaborn if you're doing brain stuff you might want to look at I mean II and that will be such that will depend on your specific use case and every all of these libraries in Python and in julia is using julia should work with will Twitter but it's it's a case-by-case basis it it will depend on your language in Python more and more the scientific community goes to our anaconda anaconda is a lot of things that are together but then after you might have a fragment if you're dealing with image that's why you have second image as I you have to keep learn these are basically hey let's get to get all of the things together that are dealing with machine learning let's get all sitting together that are dealing with images mne the same thing it's let's get together everything with seals with brains for Python for Julia I don't know I don't know enough that they have they have to see them similar things they have three ops if you're doing optimization you want to get to the up and go to the drop organization and download there okay okay so as Matias mentioned we'll be looking at Jupiter hub and how to deploy that in Google Cloud so as you're probably aware so far we've been looking at Jupiter project which itself is a server we can launch multiple notebooks that's essentially a single user environment modular and discussions it just ad but with Jupiter hub is something where you will actually have completely separate Jupiter servers isolated from each other something you typically use in our classroom setting where everyone will have their own environment and I'll just show that in a pictorial way we have Jupiter hub which is using docker swarm to create a group of Jupiter notebook servers which are all isolated from each other okay okay the Deuter jupiter hub work is funded by the berkeley data science education program just for for information what i'm going to be doing is tapping through this this website essentially a zero to Jupiter hub so I recommend you have a look at that explains quite simply how to set up Jupiter hub in Google cloud or Microsoft Azure I've not tried it myself and as a oh just on Google cloud this is pretty easy to get going okay so just to show you this is my Google account ignore those two machines that are running there it's just legend to do any tidy up so at the moment we don't have to be to have deployed at all and I will just show you I'll be working from the command line a lot so just to show you at the top there example visible I essentially have a watch which is going so if the moment we have nothing so we just can't connect to anything what this is trying to show is the number of nodes that is virtual machines available in our Google cloud and then we'll be looking at the number of pods running so this is running running Google clouds under kubernetes as docker the basic unit of execution is container with kubernetes the basic unit of execution is a pod which is essentially one or more containers or all in the same network space and then also we're showing the the proxy that we'll be using to publish our duped observer on a public IP address okay so for moments there's nothing to see there but we'll see that advance the launch and system okay okay so I said I'm going to be working from this site there are several steps to this so first of all to create a kubernetes cluster now Google Cloud then we will set up a tool called helm which is part of the kubernetes project I split the equivalent of like docker compose you would have in a docker environment with kubernetes this allows to specify in a declarative way the components of your solution you're going to deploy across a cluster of nodes and then we'll look at how we can prepare the configuration file to then deploy Jupiter hub yeah well I mean if you search already zero to Jupiter hub I'm pretty sure you'll find it it's zero to Jupiter hub I mean zero - two - Jupiter hub I mean I recommend just looking upon on Google yeah so it's actually to read the docs the bio page is the one you want to get to okay so I'll show how to set up a basic Jupiter hub installation in Google Cloud and then I'll just show how to deploy your own image so to extend the basic configuration okay so I should just say that with the Google Cloud clicking up here there's a console I used to take on pretty small up there which you can click on which will open a console I'm not going to use this I'm going to run these tools locally please just to show you that this exists the command-line tools that we use g-cloud G Cloud SDK and cube CTL are automatically available to in this environment so just to show you the there present G cloud and I want it fixed okay it's better okay just to show you in that Tim environment you have those tools available I have them installed locally on my machine that's how I prefer to work but you have that option you don't have to install any tools and to do this procedure okay so that's within my Google Cloud console okay I've actually divine going to demonstrate this I've actually scripted it up good idea that way we'll be able to go quicker and that is if you're interested that's on this type of repo mg right PyCon u.s. 2017 teary so I tighten Jupiter hub demo but it's just implementing a lot of the stuff that's on this read the docs site that's really what we should be looking at so we've seen the Google console I've already locally downloaded the G cloud SDK and the one that's available through the console and I'm going to use a command like this larger she cloud container clusters create your cluster so what we're doing here is we're creating a cluster kubernetes cluster which means that so we're putting them nodes equals three we'll create three virtual machines in the cloud okay and this will automatically instantiate that with a kubernetes cluster for us okay so this was all looking fine in my environment it's going to be bit trickier with on the screen so I hope things are going to be readable so I've got this script why I just do everything automated so the first step I'm going to run that g-cloud command that we just saw to create an environment okay so you can read that claim creating container clusters and giving it a name space named Jupiter hub namespace as a kubernetes concept and setting them their words equals three so that actually takes think about a minute or two so our studies running let's look at what comes next so once we've got our three nodes up that watch command that I showed you up here we should soon see that at least we'll start to see the nodes so these are the virtual machines of Atlas surface and so it's cube CTL get nodes step and it allows us to interrogate those I'm just running them here under watch just slice them through okay okay I'm a teenager okay so we've got our cluster and posted being set up the next step that we will do will be to download the binary for helm that said helm is a kubernetes project allows to specify the components of your solution okay and after that we'll be setting up the configuration specifically Rancho github application okay still no nodes yet so we download the helm binaries former helm in it not too concerned about those details and then we start configuring some llamó files so we can install Jupiter itself Jupiter hub itself still I go back to the cloud console up to nothing there either seems slower than about five six times this morning never failed me okay so if I may get our three nodes suddenly appeared there that's the output of our cube CTL get nodes command and if I go to the console we should see okay these three machines they're all labeled jujube dub okay those are virtual machine instances so we now have a cluster okay so the next step is our download install the helm utility fact I've already done that so just pull it down from that URL which in the home page perform the helm in it you're not concerned about the details of that we need to generate a couple of SSL keys that we will insert into a template config file okay so I'm using open SSL I've generated these two SSL keys a settle cookie SSL proxy and if we look at the the template this is config yeah Moe template where we basically got a placeholder the SSL cookie and placeholder for the SSL or proxy so substitute them in that template that's the template with the keys substituted and the next step is so now we have our configuration ready to go and invoke helm install so we we reference this URL that's not very readable it's not too important that basically there is a long URL on github the reference is what we call the helm chart okay so it's a file called Jupiter hub person with 0.3 tar.gz okay so it's basically a tower file with a set of configuration files that describe how a Jupiter hub should be installed and under helm and a qubit at this rather okay so we've launched that that was a helm install that's run okay heartily at small icing cease and this information okay so we're starting to see now pods that are being created okay so you can think of them as containers okay so we can see them in various states the proxy is in running state near the hub deployments in container crating typically after means it's pulling down an image and also it's created a public proxy which means that we have an external IP address since this address that we'll be using to connect to the jupiter console out in the google cloud but we need to wait until the pods are ready before we can access that okay looks like we ready they're both running so pick up that IP address go to my browser it's not fast okay sometimes there can be other things going on I just write yeah it'll come so whilst waiting for that to come up you know if you think about what we've done so far it's just with one command we've created a cluster of three virtual machines in the cloud and then we've just modified a template describing our configuration with a cup of SSL tokens and with helm we've actually deployed the Jupiter so here it is now operational as you can see very few commands we were able to get Jupiter hub up and running as matias mentor that mentioned earlier of course this is so far this completely unsecured jupiter hub we're using HTTP and there's no real user password authentication here we can just enter what we want as a user and password on that zero to jupiter hub read the doc site they describe how to go further to do Google authentication for example I'm not going to do that other things I will do so okay it's login as user a very imaginative we can see there is already got a notebook I jump to go back here and just show that I logged in as user a we have here a new pod Jupiter - a - one so I then add another login again as Mike for example we get Duke to - Mike - - and so on okay so we got I haven't got much so far Mincy typical Jupiter server we see you've got Python three we can start the terminal and stuff early classic stuff okay so so far what we've done actually I could have a look at that is that that one have you logged into my sir I didn't create our second bother some other okay okay it's not very reasonable readable I'm afraid of the screen size but what I'm doing here is I do a cube CTL describe of the pod which is one of those notebooks so I'm going to do the Y created which is Jupiter a - one so I do a cube CTL describe it show me details and it's just to show that we can see the actual docker image that were deployed by default with this chart that we've deployed okay so an interesting next step would be to change that image put a fairly basic notebook with item three let's see how we can start adding things in okay so what I propose is going to take a doc file and we're going to derive an image from Jupiter based notebook this is one of the notebooks defined in jitter then due to the darkest acts okay and we're going to going to create just a base notebook and we're going to just copy in the ipython chip git repository that we've been using in the labs today okay so this is a docker file from do to the Jupiter base Network I'm just setting up the work there for user jovian and I'm copying in just a local copy of the ice so I type in in-depth repository so I did a docker build on that i doing it on my system with my username gone MJ right up to the bottom left so we built that container and now we're going to push it up to docker hub on the 0 to tube the hub they they use the Google container registry which you can do I I chose to go with docker hub okay and as I've been doing this all morning so there's only just that one layer but it has to push up to do docker hub okay so I've got my daugher image out in docker hub now I want to add a section to my config llamó the file that's going to tell him how to configure I Jupiter hub and I'm going to add this section referencing my image in doc hub okay so I now do a helm upgrade previously we did a hell install with Juba dub and with our config ml file now we simply do a hell upgrade and we should see things start to happen up here okay so you can see we have some fact we have three pods called pull latest something so this is clearly a pod one per node which is pulling whatever image layers are necessary for a new service by experience when these have terminated which is fairly quick I think we then get a change here we have to wait a while the new hub deployment - to be ready okay so the old hub deployment we had which is terminating and we have the new one which is creating I think that takes another minute or so until that's running so it's interesting to see we've changed the base image that will be used when we launch new notebook servers okay new Jupiter servers so if we log out and log in again sessions will be using that new image but the the current sessions we already have continue to run all the same with the old image so what I'll do after this next step is that I'll do another just example of modifying the image I'll go for a larger notebook image not much larger about that bass notebook is about 280 megabytes I think I'm going to take a minimal notebook which is about nine hundred megabytes okay so I hope deployment is running now so I can go back here I can log out the low in this mic change so we see Jupiter might 3 now new session and if we look here already if I do so I do a another cube CTL described pod just so that we can see that we're now using this new image this pulled off github but much more obvious is when we actually go okay when we go to the the notebook yourself the server we see that we have the ipython re that can maybe see we have the ipython height and depth stuff copied in okay okay so just an example of customizing customizing image setting having stuff N and we go to a third example so I said I'll go to a minimal notebook image this is something it's nine hundred megabytes personally I like bash kernels with Jupiter I do a lot of stuff with bash and the Jupiter I do docker tutorials and stuff so I yesterday were doing stuff some stuff in Python sadaqa kubernetes amaze off and some other stuff in command line for docker communities and made off I'm a find cute Jupiter really useful for that for documenting a lab and you can run it from start to finish check that everything's still working so I'm a real fan of the bash channel so here's a docker file for this third indication so you don't see the first line here those are from minimal notebook set at my work dear I have a requirements file which just specifies bash kernel there's a command to do about the pip install that requirements file and then a command to do to actually install the bash kernel and I copy in just a skeleton example of some sort of basher A's tutorial sister give us some meat so build that new image so build a new image we'll track it as we did before and I'll push it up to da card okay a new image is ready so we'll push it up to docker hub go very quickly again now we modify again completely ml to reference this new image which is in docker hub and then we do he'll upgrade again without we have modified completely mo so you can see now I'm referencing this new image and J bright with my mid notebook it's not the same one as we're running before okay so that's going to take probably a couple of minutes yet that too to deploy we've got those three pool pods still running and after they've run it will roll over to the new deployment let me just mention in that zero to Jupiter hub read the dock site the mentioned tool called source to image part of the openshift project which essentially rotates kubernetes distribution looks quite interesting you can do the same sorts of things I'm doing explicitly the docker files so let me let me go to an example somewhere so I actually tried it in the last hour or so and failed but first time I use it it's interesting to see they exist the basic idea this is called s2 i source to image you can download the binary for various platforms and essentially you do STI build you can buy github repo where the source is for for the image you want to build so but I wanted to use the git repo Jupiter doc again dokodemo images last it builds the image which is used for tried on Jupiter dog I've always liked that multi-language demo age so you provide the URL of the source you provide the name of the base image so those are the dockerfile air doesn't actually take the base image that's in docker file you actually override up and then you give the rest of the google container registry to upload your built image okay so yeah I tried this earlier on the dock at any moment it was swelling some reason it was a first attempt so I just tell you the tool exists and it looks pretty interesting with a look okay I expect this is finished to deploy now yep I don't contain the crating should take another 30 seconds or so to complete yes so it's one in fact one pod which probably has one container and that container is Jupiter observer in a container Jupiter server sorry they're contained out not to be down yeah if you log in yep okay we're coming to the end of the session I'll come back to that as it completes other questions about what I've shown are any of you deploying Jupiter hub already maybe in a classroom environment or one - and I think it's pretty cool that they've made the zero to Jupiter hub site available I know last year I presented at Europe - in fact this this tutorial more or less but not at all magic happens quite a few people interested in how to deploy Jupiter hub which I think was a more difficult task a year ago it seems to make it pretty easy thing to do okay that's running I am do a logout that one in like 92 I mean it's you know the equivalent of what we saw before this timer just modified so we now have in the kernels we have bash kernel available and I have an example notebook which in theory is a tutorial on batteries so I can bash deserve some love okay so the first thing I do is just some magic to see what system we're running on so this is actually a Debian Linux and the bash version we're running full at 3.30 I mean if there was a real bash to Toria your them to know what new version of the OS what version of a bash you're running and then the content of that is not too important it's not so even finished actually should go back to and finish that sometime okay so I just wanted to show you how easy it is to deploy Jupiter hub in your own Google Cloud account and then how easy it is to to modify the images that will be presented to users so I encourage you to look at the 0 to Jupiter site remembering of course this is not a secure implementation in the site they describe how to do proper authentication which is something you would definitely want to do session a classroom environment okay it's time for any questions I'm available [Applause] while it will have its own copy her files yeah yeah iterated you have ways of mounting things everything everything is not

Channel: PyCon 2017

Views: 21,472

Keywords:

Id: VQBZ2MqWBZI

Length: 199min 6sec (11946 seconds)

Published: Fri May 19 2017