James A Bednar: Panel: Dashboards for PyData | PyData Austin 2019

so before we get started can people have a show of hands how many people are consider themselves savvy Python people like ready to use Python happy about it I mean people consider themselves kind of Python newbies used to some other language but trying it out okay how many people are well versed in the SyFy and Python for data science stack like you use all the tools use pandas you stop okay how many people have used map plot live either pandas plot or however okay all right that's a good start if anyone is sitting next to somebody who raised themselves raise their hand has a relative newbie you could just raise your hands again if you're pretty new to Python if you're sitting next to a person like that help them out please with me they are running into problems you're the expert I'm Jim Bednar I'll be presenting this at least unless my voice gives out this is my colleague jean-luc Stevens we're both from anaconda which is local we've seen lots anaconda people around here and former anaconda people who are now starting their own companies and such it's kind of cool all right so the post-it notes are for your laptop please put a green one on there if you have anything running if you've followed the instructions and you have something running either locally or by clicking on the cloud thing everyone should have a unique user name and password for the cloud offering and so you shouldn't be stepping on each other's toes unless I made a typo but you are all still sharing the same machines so there's a big there are a couple of 40 node clusters that these accounts are on so if some if everybody's running something enormous at the same time it is possible you'll step on each other's toes probably not we're not being very ambitious in this talk but just be aware it's a shared resource and of course Wi-Fi is a shared resource so everybody is kinda installing an environment at the same time you might be fighting each other for bandwidth okay so put your red post-it note up if you're having problems following the instructions if you're just working hard and haven't finished it then no post-it note green if you're done if it works because I want to see a lot of greens before I get heavily into anything okay okay if I fail to spot a red one right wave it at me and I'll send don't look over to help you out okay all right so to make you so good use of the time I'll start but I won't be assuming everyone's ready to follow along so I'll just be showing off some stuff at the start so don't worry if you're not done setting up keep setting up that's your main goal to get something running in fact that's what I want you to walk out of here today with there are emissions are low we want you to have run something they have successfully running that is the only goal here really well to be exposed to some other stuff but to have run something yourself so if you're not able to set it up and it's not working that's something to work on and we can help you with that all right so this is the basic structure of the talk can people see this in the back of the room it's bright enough okay thank you so we're in the setup phase right now then I'll have a background and demos kind of show you various examples of things that you will not be able to do during this session they're more ambitious than that and then we'll start out for what's kind of a one-liner make a dashboard with one line of code and you can play around with that for a while then we'll say okay step back and do it a little more step-by-step and then be more ambitious so it's okay if you get to here and aren't able to do the further exercises at least you'll have done something if you're failing it exercise one pull somebody in talk to your neighbor you hopefully you get something working at least alright if you've got something running you can run this first let me restart and clear everything I suppose I should have asked how many people have used a Jupiter notebook how many people have not used a Jupiter notebook great I will fail to explain lots of things then okay so the first cell in this notebook is just to make sure things are installed properly all it is doing is checking the versions of panel pandas matplotlib okay and Holle abuse if you if it succeeds you'll see nothing if you see error messages something went horribly wrong and it's worth figuring it out alright so at the beginning I'll just give you an idea of what the heck panel is first off go to the website panel is an app and dashboard development system for Python now apps and dashboards what I mean by that is basically a website but it's a website that you write in Python it's a website that you can easily have data and plots and all of your PI data basically everything that's that's in Python and as data related should be easily able to be turned into a website it's to be turned into something you can share with people and turn into something interactive in Python now obviously you could go and write a website in HTML and in JavaScript and CSS if that's your thing go for it bye this is for python for people who are comfortable in Python and want to stay in Python but still want their work to have an impact want to be able to share it to other people who aren't necessarily gung-ho about Python but you still want to make an impact on those people or just for your own purposes you just want to be able to set up something interactive to play with yourself maybe I shouldn't said that that way anyway alright so back to the task at hand panel is a open source library developed by anaconda and we developed it as part of our services group the services group has contracts with government agencies and with companies all over the world and basically we listen to their problems and then we tell them an open-source way to solve their problems and suddenly their problems over and over we're about how to build a dashboard and how to empower their day scientists to make an impact on the decision-makers in the company and previously what people but people told us over and over is I've got my Jupiter notebook it looks great and then I hand it over to the JavaScript team and six months later they make an app and it isn't right and we work and then eventually it's sort of okay and meanwhile the business has moved on that is not how things are supposed to work you're supposed to be able to be a data scientist you discover amazing things you find cool transit you want to explore them and then you want to share it with people are going to make a decision fund that can that project do this whatever you need to make decisions so we over and over kept hearing from people who had trouble linking up their cool data scientists to actually justify their salaries for having an impact on the organization so we listed we kept hearing this over and over and then we said okay we're gonna break down we're gonna build this as an app and we're gonna build some library that makes it easy to connect up something you got working in Jupiter to being something you can deploy now at the same time we're building this other people started building other things voilá has anyone heard of that that is a sort of competing tool for some of the things and all kind of point out where it word overlaps and where it competes voila let's you take a different notebook and deploy it as well it doesn't have any special features to make that easy but it will it can take a jupiter notebook and lets you deploy that panel can be used whipped together with voila but we're gonna focus on talking about panel here because we can't really control that now there's also a package called dash dash is a little different - doesn't take do for notebooks - just for building dashboards in Python but only if you're building a dashboard where panel does is try to say you're working with the jupiter notebook you make something you add a few widgets you make it a little bit interactive and you gradually build up an app and eventually you build up a dashboard eventually you build up a multi-page dash for it eventually becomes somehow a complicated system but you never had to switch tools along the way so - is not like that - is okay I did some stuff now I'm gonna build a dashboard in - throwing away all of that voilá is more similar to panel in that way you have notebook and you can deploy it so voila and panel are more similar all right so here's just some random examples these are all taken from the links here examples up by biz or the panel gallery this is the panel gallery and I've already opened some hopefully they're still running here just to clarify there are two types of deployed panel apps one is where there's Python running so you click on things you're in a web browser something gets eventually transferred to a Python server some computation happens and you see the result that's a Python backed app and if you go to the panel website some of the things you'll see there are Python back other ones were written in Python but the final result is has no Python in it it's just some deployed HTML Javascript and whatever I think one of these is okay here this one is an example of a deployed app here that it whoa that is only JavaScript so this thing is written in Python and usually this is panel code you'll get to that the final result is a is an app with widgets and so on is all connected up but the result is in JavaScript hopefully that doesn't hurt you in any way basically you write a specification most of these are declarative it's declaring what you want the exilim indentation of it is in JavaScript so most of the things you'll see are different they require like this one requires a running Python process and so if you've not visited the page in a while as I haven't I have to reload and I'll start what's called a session I'm here in my browser if I drag something here it will send a command over to the Python process some computation will happen things will come back and it gets updated in the browser that's a Python back system so this Python back system in this case is drawing some pretty pictures basically this particular one okay yeah this one is deployed on the cloud there's some ATO AWS server that's burning some money as I do this my employer pays so whereas this the other one I showed you here that's all happening inside the browser so there's some JavaScript code eventually it just gets sandal it in the browser at the computations happening in the browser there are pros and cons if you have a Python server you know anything you can basically any possible computation you can write in Python you can have something on that page that triggers that and you get the result and if you've made it into something that happens only in the royal tour then you can easily just email it to people so they're frozen cons will show you both see this is one that was well it was running so this is just something to help you explore a little mathematical thing yeah it has one widget you get to choose what your function field is that one's a boring one and then you can look around whenever you click something happens in the browser it selects your mouse click there's out where you are in this plot sends it to Python Python computes a function plots it goes to the browser okay so this is this one's Python backed this one couldn't be done in the browser because this is 213 glaciers in the world you can select on each of one of these things and all these plots are interlinked this particular plot is done by program called data shader that turns data into images and that all happens at the Python side the server side the image goes into the browser so the browser doesn't crash even though there are millions and millions well in this case only 213 but this could barely probably work in our web browsers but it can work with millions this particular one is a billion GPS locations here this couldn't happen in the browser so it advised them in here what's happening is it sent a request to fight on toll Python to render it and now it's going to re render you can actually see the delay with Python here because this is a huge data set it's just on my laptop do you see what's happening there so this is the sort of thing that you can only really do with Python in JavaScript because the browser's model of how to manage memory does not involve a billion points it will never as far as I know involves a very small sort of thing sandbox for me every other browser tab but if I find can handle whatever size of data you want okay one thing I have well this this shows a little bit of one thing I don't have a good example of we've tentacle created a lot of these for our clients and then we can share them a complex multi-page apps they're hard to share as examples this particular one is not one that we've made for a client so we could share that it but it's not also it's also not very multi-page it has a few tabs but you can actually use these tools to create things that are whole websites and have state that persists across multiple pages and such that's much more complicated but just so you know there's Headroom you can build quite complex things all still in Python okay I think I'm done with demos and ready to move on to actual work so I do not see any red on anything so I see some things without a green post-it if you have something running of any sort please put a green post-it on your machine if you're having problems and have nothing running please wave a red post-it and make sure we know okay I guess we're ready to get started before we do does anyone have any questions so far just see if you're in the right place or have some basic a question yes [Music] yep yep yep well we'll get to that there are a lot of a lot of options one of the simplest ways is if you have something that has let's just put up something here that has a widget something that is very general that could work in all cases but you'll immediately see the limitations is if no matter what is here this is a function of these values now what is the if you had one slider and it had ten values you would just need to export to the browser ten plots to handle one slider but what if this one has ten sliders and this this one has ten and this also has ten in this and so on that space of possible things to export to the browser well very general you can handle anything it'll immediately become a very large thing that you're trying to export to the browser because you're essentially pre-computing everything you might ever ask for in python getting it down to the final javascript HTML static stuff and shipping all that to the browser that is always going to work but it's in technical terms but it may eventually it'll quickly overwhelm your browser so there are other things you can do which involve figuring out a way to link this using javascript to this you write the two lines of JavaScript and it connects up directly and there are other things where we've already done that for you for simple things like having something to control the color map that just has already been solved in JavaScript so we can do it in Python but we also hand through in a little thing where we did that connecting up directly and but in JavaScript so some so there's a complicated answer basically in Python you have can do anything you want some subset of those will immediately map directly to JavaScript being done even if you can't have that you can do this exhaustively collect all the data and do that that's also automatic if you want anything more ambitious you're gonna have to do a little work so basically if you can count on Python being there you can avoid all the work you're gonna stay in Python do some Python get some app and if you can count on there being Python to deliver it you're done beyond that you do more work yes sure I guess the underlying panel works with a ton of underlying libraries and as long as the underlying libraries have ways to generate SVG typically than you then turn into PDF PDF that can work you can also just hit print on a particular page in your browser and that often works but it's not intended if you take a step back the underlying Jupiter notebook is usually easier to turn into a PDF so behind this thing is a notebook with maybe twelve lines of code that notebook is much better set up to be able to turn directly into PDF so typically you'll have a workflow where if you want to have a report you have a notebook and you spit out a report that's based on that turn that notebook into a PDF and then you can also turn that notebook into a live running dashboard but we haven't typically turned it into a dashboard and into a PDF because basically this is waiting for a user to do something whereas the notebook can laid out all the choices I'm having been made automatically without a user and one final point is that all the panel Doc's that you'll see online those are all generate it from notebooks turned into HTML and so once those are in your browser you of course you can save those to PDF there so that's a notebook to browser to PDF workflow but good question anything else before we start okay let's actually run something so I cleared my notebook here if you've already been playing around you might want to clear your notebook again like I said this is a not very ambitious set of data here it's not a large data set as not a complicated data set this is meant to get used to give you some success so this particular data set we're reading it's appropriate here as it gets a little hot is measuring the characteristics of a meeting room the temperature in Celsius I guess humidity how many people that might have to just be whether it's occupied and over a series of time points so it's basically somebody instrumented a meeting rooms climate control put that out as a standard data set so it's in a pandas dataframe panel doesn't care what type of data hey you have whether it's a panda data frame it cares that it's something that can be visualized particularly if it can be visualized in a Jupiter notebook so this particular one you can see it's got some columns and now let's say we've figured out a way to plot it in this particular one we wrapped it up as a function because we're going to want to call this over and over but we wrote two functions here one is basically the idea is that we're going to smooth want to pick one of these columns smooth the values slightly and then plot it so here we write a function at the moment it's using matplotlib but we'll use other plotting libraries and it's given two sets of data one is the time series and one is a subset of that time series that we'll use to highlight it you'll you'll see in a minute and also we're going to we have a function that's that will try to compute given a time series and a smooth time series find outliers find things that are very far from the smooth value what we're doing here actually mit doesn't matter at all I'm just trying to explain it just think of it as something that makes a plot okay so here's here we're calling this function we're looking at the temperature oh we've plotted the temperature so that's the the raw temperature and then smooth it with a running average window of size 20 data points and we're gonna highlight all the data points that are I guess 10 Sigma's away from the smooth mean and so what that does is here this is it's basically highlighting areas of shark change and if we want to mess around with that we can change that how far away do you have to be to be highlighted if you have us if you have some distribution you don't have to beat in sigma's away you only have to be five you'll get more data points highlighted right and eventually you highlight everything okay you can keep doing this if you're a Jupiter programmer and Python programmer you can have a function like this you can explore this parameter space you can do that already you don't need panel but it's awkward and painful I mean but - it's you have to keep you don't know what the allowed values of Sigma are you don't know what the allowed values of window are you just type it and you study the code and so on so it takes some Python knowledge to work with that even for this totally trivial function plus if we wanted to share this with our boss and we wanted the boss to understand oh okay this is the parameter then you need to watch for this is what we're trying to optimize for they're not gonna be happy with this so we're trying to immediately get you to be successful so we give you a one-liner this is I mentioned voila voila can use I play widgets I play what just has a function called interact you can do the same thing we're showing here roughly the same syntax to use with I pi widgets and voila so up to this point you can use whatever library you feel like this is not know this is panel panel is PN so panel dot interact it's just copying the I pi widgets API so now if we do this now we have this function and we you notice that we haven't immediately got two widgets and if we drag the widgets we can see what the effect of Sigma is and we can see what the effect of the rolling window is without with essentially zero work yeah so let's see how that works so that this is the magic function so how does the magic work the magic won't always work so we got to understand the magic okay so what's the magic this find outliers function that's what we're calling with interact and it has one two three four four arguments why didn't we get four widgets because the magic only goes so far look at the widgets we got we got one for window and one for Sigma what other what other widgets do we need oh well there's a variable which selects a column in the data set the Interac doesn't have any idea what the columns in your data said it doesn't know that there is a data set it has no way to know if the value defaults to temperature that it could be anything else those could be any ASCII string right or Unicode and what about this one view fun well that takes this function here how does it know what possible functions you could get it doesn't so what interact does is to make it be useful it does what it can okay I'll I'll deal with it I ignore the things I don't know about I'll give you widgets' with the things I know about I cannot help you with those other widgets it looks at the default values that's correct so if I had changed that to a float then the it would be a floating slider yes very good question what about this window function doh it's gonna needed to be a window can a window be negative it cannot be negative can this widget be negative it sure can it doesn't know window it doesn't understand that window is a has any physical meaning it just came up with a widget for you it doesn't know it's magic but the magic only goes so far so exactly so you can though we'll get to that in just a second so basically the magic gets you one step and then if the magic if that's all you need it you're done you go on with your life you have success great took you one second to try it out and it's trying to design to fail gracefully so that the things that doesn't know about it just doesn't do the things that it does wrong well if you don't want it to be negative don't drag it that way okay so it does what it can it's true it's a really helpful kind of stupid friend all right so what if we break this down though we can actually do a lot more with this when the magic isn't enough like what if we didn't want our widgets to be bubble what if we wanted to be there to the right what if we only wanted one of these widgets we didn't want one for Sigma my boss doesn't know stigma that sounds a Greek Oh No so you want to hide that okay well now we can look into what the magic is actually doing and we can help put the right values for the window so we'll get to that so first we'll dress doubles question which is well let me just go ahead and show that it works first whoops lower case okay first you'll see first let's see what we we've got and then let's see how we got it what we got now is two things we got window to start at one physically realizable a window can be one or higher okay so how do we get that well we got that by telling passing in an extra keyword argument to the interact function saying the range of window okay so we used our human knowledge of understanding of windowing and we're able to express that for the interact function okay your magic is not enough here's a little hint so we gave it one hint okay what else do we get here we also got this cool drop-down then it allows us to select the different columns in the data set how do we get that looking back at the code we again gave a keyword argument and we told the told it the possible values of variable we gave it a list of the columns in the data set see that now it doesn't have to do magic it knows what variable can be and it lets you choose those okay so you're helping the magic alone and the other thing that's happening here is that we can actually oh we also told Sigma to be in a more reasonable range I don't remember what it was before oh it could have also been negative so Sigma can't be negative so we told it that all right so we helped out the magic a little bit we also printed this object so if we call interact we get the the result displays automatically so if we just do this sorry if we just call it it displays automatically but what if we try to grab a handle on it and print what we've got instead then we can see what was returned from interact is not a monolith it's monolithic thing it is actually a compositional thing it is a column composed of a column and a row okay if you look back at the original you can see it is a column and in this one this bit is a column this one is a row with one item in it so okay and then what's in there what's in the column a selection widget a integer slider and an integer slider and what's in the other one you might put lib plot okay so this now that we've got those we realized that this magic bundle that is dropped on us it's not a dead end we can actually work with that so let's let's work with it let's take it reorganize it everything so now I said no the widgets were up here I don't want them to be up there I'll put them on the right I'll put a label I'll tell my boss what to do how did I do that I took the original structure here and I indexed into it I'm going to now make a panel row the consists of item 1 0 which is the plot item 1 0 so this is just list accessors so we're using lists accessors to access this this is a list of lists or a list of lists of then it can be a list of lists of lists of lists however many you want so you just access something in this compass it data structure and then make a column over here consisting of some text which is actually in markdown that's how we got the bold it assumes by default Texas markdown and then we put one widget in another widget and we ignored the Sigma widget see how that works and using this technique we call this approach shortcuts not dead ends we give you a quick shortcut interact but it is not a dead end it is a shortcut it gets you this way and if you want to go over here you just do a little bit more work as opposed to a dead end which is here's the magic you know like magic go install tableau so we don't want that answer all right so where are we alright so we were able to come up with this non dead end thing where we could make our own thing also you can see that if you really want to care about the details if you still want to mess with the the Sigma there is nothing special about it being shown together these things are reactive objects that if you display a widget anywhere will it will still update it and it also is a it's available for you in Python as well if you want to mess with it yourself it will update maybe you see how the plot is updating even though I'm just fiddling with this you can watch the widget if you're not blinking if you do this you'll see the widget value change right so basically this composite objects you got is totally not add it in because you can pull it out you can put things around you can mess with it in Python you can set up a callback function that messes with it you do whatever you want so you've been by magic given this bundle that is probably often useful and it's not a dead end you can do what you like alright so now it's time for you to play so exercise one is pretty freeform and they get more freeform after that exercise one is to do something right some function that does something if you're having trouble with matplotlib or making plots just make it return a number make it return some text whatever you can panel if you just write P and panel and give it something like 45 it'll it'll whatever it is it'll display the thing and if it's a mad lib pot it will display a plot if the bouquet plot it'll display the plot if it's the SVG or a PNG or a JPEG it'll display that thing so get something you might want to display like figure out and think of anything if you want some inspiration click on this link you can see lots of things like equations or pngs or different plotting libraries YouTube videos I don't know whatever you want you can at the moment stick to the pains section we'll get to widgets and exercise to you so the goal is to get something that displays on its own and then to write a function that has some arguments and does something so let's just do some live coding of the sort that you might want to do return Arg x - okay and then try this so what happens if you do in our equals one okay so here's here's a panel I just created see how exciting it is it's got a widget I drag it something changes you do that to make the thing make sure function return and strings or numbers or something write some function with some number of arguments and then use interact to make it an app yeah and call one of us over to help you wave a red flag or whatever if you totally have no idea what you're doing or you've hit a roadblock and you really want to find out the answer yes in this case it's realized that Arg is an integer by inspecting the default value and creates an integer slider because it's a default value it realizes that the thing it's returning is a string because it checked return to string and so we called it once with the default arguments found that it returned to string and that's what's displayed it should be able to return if should be okay even if it's changing type all the time but we should try that yeah so it doesn't have to it just displays whatever comes back from that whether it's an integer as it is in this case or it's a string in these other cases that make sense so essentially you've just got a function and panel knee you need to make sure the function has a default value or panel can't figure out what what the what the allowed values are and it needs to return something that could be displayed whether that's a string and integer or image or whatever that's the only requirement here and so basically try to get something to work if it doesn't work talk to your neighbors or call us over if it does work then think about it am I done is this function gonna work like what if every time I drag that slider it takes ten minutes what am I gonna do that so try to understand the limitations of what you did because that leads to everything else or the thing you've got here what if I need to click on that and change the font of it how can I do that I mean try to come up with something that is not solved as well as whatever you've solved try to figure out a need because that's where we're going after this and I'm checking time now looking good so I'll give you four minutes and then I'll ask to see how far along people are you can keep working at that point if a lot of people are still heavily into it I'm J Bednar at github if anybody is wondering so you were trying to read my badge so David NAR yes memos in many cases we haven't published a generic interface for that we would love our contribution for that akin to what's in streamlet I should have mentioned streamlet as a new competitor there's streamlet wallah and panel or basically the end - those are the four current hot libraries for dashboarding streamlet takes a different approach which is to memorize essentially everything all the time because it reruns your entire code every time you drag a widget so you if you're gonna do that you better be memorizing some things to cache them whereas panel allows every which--it to be tied in in the specific independent computation and so it doesn't come up as often but it's certainly it can be useful and we who support it in some cases Stream Li T streamlet lets you use your own browser streamlets really don't like notebooks because it lets use your own browser your own editor your own full IDE it have all syntax highlighting and everything refactoring whatever you is in your editor and every time you hit save streamlet will re-evaluate that file having cached results along the way and update what's on the screen so stream that helps you work with an independent file the caching helps there and it also lets you write something that where they come to the as you drag widgets it's cached streamlet focuses on the low end of what you can do it makes it super easy to put some widgets and put something on there and then in it hasn't been around very long and it kind of has a dead end like oh wow that looks cool that was great what if I want no so it can it has it's as a really strong case for a certain type of usage and a few years from now it may expand to to handle the types of things a panel can do but panel was built on bouquet which has been in development for since 2012 I guess it's been it is a mature and established library panel is new panel is sort of easy to use features on top of and support for other libraries the book I didn't have panel is saying okay building on what's in bouquet which is rock solid been there a long time and really covers all sorts of complicated situations that people have any thought of yet on that basis can we build something that's much easier for a data scientist to use something that works well in Jupiter notebook something that makes it easy to make apps right away so that's what panel is so it's built on this deep and wide foundation stream millet is there's no default foundation it is exactly what it is today and it could could have a great future so it's fun to be this is a really good time to be trying to do dashboarding in Python because we've got four power players all watching the blog posts from each other and say hey I could do that and battling in the blog posts yes oh you wait till the next section good question okay is anybody so deep that they can't be bear to be pulled out of it okay and so show of hands how many people successfully did anything all right excellent that's actually uh you can go home I mean anything else is a bonus as I said my ambitions are low if you want to have high ambitions there's a YouTube video of me at sci-fi this summer three hours long being much more ambitious if you can handle listening to me for three hours you go from zero to very complicated big data Geographic applications whatever but that is much that takes your head getting your head around and takes kind of buying our entire stack panels one of the things we work on we work on a whole bunch of things and that whole group is called hollow viz panels one of the products there but it's the one that reaches the most people I say products this is all open source we don't make any non open source all right so we're going from there that was exercise one you did something how many people saw a limitation like something like this you know one was just pointed out there what if I want my widgets to depend on each other like if this one might be a maximum value for this other widget that's a limitation I'll tell you how to address that anyone else see a limitation are like like oh looks like it can work but what about this anything like that yes yeah that's right so matplotlib and jupiter notebooks have a long messed-up history of how it interacts with the notebook how it displays saying the typical thing about plot libous use MATLAB in line and then it doesn't then your figure doesn't get returned like it normally if you take something at at Python prompt and just write seven seven is returning it's displayed you put a string there the string is displayed but if you do a matplotlib figure in line what's returned to some kind of garbled meaningless text and then it also displays your plot panel requires whatever you do it has to be returned because otherwise panel can't get ahold of it you might see on the screen but panel can't get at it so you have to make sure that when so the code up here from Apple lib is specifically making sure that it returns a figure not just displays a figure and that's what makes everything else work so you need a function that returns something that is displayable and then interact will work that is your one requirement and matplotlib and jupiter could have been written to do that way they met the Jupiter interface term applet lib ten years ago could have been written that way it was not it was written to display it as a side effect inline in I don't know who wrote it I don't want to meet this person is weird but it can be made to behave quite reasonably there's nothing wrong with Matt but lib it's just about the default interaction when you use inline in in the notebook so don't do that do this yes correct yeah it wouldn't confuse panel it might confuse interact in which case you would need to give interact a hint just like you do for displaying the ranges you can also do a hint that tells it don't pull out try to plot that thing it will almost certain in normal cases it will ignore the things that doesn't understand if it finds something and you've realized oh it's paying it into that very I really want to ignore it you can pass a little hint it has never come up for me so I don't remember the syntax but it there is code to do that and that's the same for interact in I buy widgets it's got all sorts of ways to give it hints okay any other limitations or problems or you're anticipating I'm gonna focus on a huge class of limitations which is essentially what happens when I drag this slider what happens when I drag it is it calls find outliers and it generates something and displays it well what if that function takes more than a second if it takes five seconds as I drag this you can easily start to lag behind what I want if it takes five minutes you better not drag that thing you might want to click like that and then we come back you see that now those doesn't mean you can use it you can't use it for functions to take five minutes you totally can but if you use interact you're gonna have a not very usable display because any change to anything will result in the entire set of computation being redone so interact is a great way to get started if your needs are modest but if you want to do something complex like have these interact in some way as was mentioned you can't use interact and if you can't accept that this one function is going to be called every time you do anything in the GUI then you're gonna have to do something more complex just as an example the more complex one I don't know if they're still running here yeah so this is a more complex one and what's happening here is to regenerate this whole data would require iterating through a billion points yet here I'm dragging this slider and it's immediately updating this kind of cool mountain range thing how is that working and I can turn off the the data as well that's all immediate the reason that's working is that this is actually a pipeline of operations and these sliders only affect the very last step the data doesn't have to be rear-ended for me to change its opacity that is happening at the very last step in the browser and I can actually change the underlying plot underneath that that also doesn't require rendering the data because this is an overlay of data on top of a map I changed the map I don't have to revisit the data so each one of these things I can change the color map of something immediately doesn't have to read and write the data but if I wanted to change what I'm computing that's going to regenerate the data and probably you can't see cuz I chose a weird color map but in fact it's still computing it's gonna take a while we'll take five minutes but that one is going back to the original data in real reallocating everything and so this is a panel app where each one of these things has been very specifically tied to the computation that the disease so you can get the fast things fast the slow things are slow because they're slow but you don't have to get this little things if you don't need to okay you can do whatever you want this is a column of widgets it can be a column of widgets with text there one thing I don't think maybe zhun look and correct me on this I don't think we've ever implemented the hover text for each one of these it is supposed to be there we already have ways to declare it but Bucky didn't make it easy to to make the hover appear but you should be able to every time you have some variable have associated ticks that hovers if that doesn't work right now then it's we just have to make the right person listen to us yes display the error the error will often go off into lala land depending on how you created this if it's if you made it in a do promote book this particular one is in a different note book that was saved to disk and then it was deployed at a command line and so on that command line there will be messages printed and also there are messages printed here in the JavaScript console in this case there are no errors so I'm happy but if you look at your browser's JavaScript console some errors will go there if they're generated at the JavaScript side so if I were doing something here and there were some coding error and it realized that some variables undefined that will show up in your brawls or browsers JavaScript console it depends on your browser how to get to that this particular one it's under more tools developer tools for Chrome but each browser has its own way to get to the JavaScript console yes pipeline so we're we're wearing outside a panel for this question but it's a good question panel is happily displaying stuff but what it's displaying can be arbitrarily complex and in this particular app which I'm reloading and it's waiting on but for this particular app it's using a program called data shader data shader it takes datasets of various types renders them as a fixed sized array and paths and that array is only what's sent into the browser so the way that we can have a million billion data points here that we had earlier was a billion data points are in Python some memory or even not it can work out of core using desk the data points can be on your disk it aggregates them renders of them into something a fixed size array that something gets put into a binary chunk of data that binary chunk of data is sent to the to JavaScript and the browser the JavaScript and the browser scales and puts that on the screen and then you see it updated so at no point is the data ever in your browser with this program it is a rasterizer so it's a handy one because it lets you be independent of the capabilities of your browser and python you can do that but right now it's just using the default which is that when okay when you interact with with this plot to drag it around or whatever bouquet realizes that the range that needs to be plotted has changed bouquet asks Python okay and I'm out of date give me the new stuff you weights you still see the old stuff you saw it earlier anyway so it has the new stuff okay got it put it on here so there's the the data is actually not stored anywhere it's just requested by the browser the browser's in charge you the user are in charge us your requests if pythons not there to satisfy it nothing ever happens for instance on our website if you just go to data shader at org and try out some of the plots they might or might not actually do anything you go and zoom in on this thing well it's just in the browser it does an update because there's no Python this is just a website it's just sitting there with a static export of my file javascript is happy it does the stuff it computes the new range it says python give it to me and no one ever answers it's very sad actually but that's how the website can work so the data shader website has no Python backing unless you click at this very top then you can get a running version of it when you click on that a Python process is launched and or well there's a Python process sitting there a session is launched on that on that process so in this this is what you see in a static export of something that uses data shader is that nothing pythonic ever happens okay anyway that's data shader another in our family of tools and it's using a program called Holub you used to do it another in our family of tools I'm here to talk about panel so back to panel all right we're worried we're finished exercise one okay we ran up against a limitation of of interact was is my conceit that we played around with interact and with whether you did or did not just imagine at some point you hit a wall you can't force things in to interact you added some hints you added some more hints but you want to do something that doesn't really fit with interact find you can drop down a level and then work was widgets directly so if you do that you can do the same stuff you did before but here instead of using magic of interact we explicitly say I want a widget in fact I want two widgets first you create a couple of widgets here one of them is an insulator just like you saw before but in this case I didn't want that drop-down menu I wanted one with with these things you can actually give that as a hint to the interact but let's pretend you couldn't let's explicitly make a widget like that now if we did that and we just look at that one widget okay I've got a widget it doesn't do anything but I've got a widget and the same for this other widget so you can just make all the widget you want they won't do anything until you do something with them so this is just a way to explicitly start making widgets so let's say we made a couple of widgets all right and then having made a couple of widgets and you can see at the panel website all the widgets that are available there are lots of options all of it originally published by bouquet so let's say we made some widgets now how do we get to where we were before with interact it's actually quite like what we're doing before what we do but here instead of saying I'm going to magically inspect this function and make this blob here we make some widgets and you're going to decorate the function and connect up the widgets to the function how many people know what a Python decorator is how many people do not okay a Python decorator for the people who do not the typical thing you use it for is to take a function and transform it into another function so you've got some function you decorate it that little @ symbol means to decorate that function so that this line doesn't do anything on its own that line applies to this one and it transforms this function that takes a string and a an integer and it transforms it into something that can take widgets that's what this does it expresses a dependency between some some arguments and some widgets makes sense so here we're declaring that this argument which is a string actually should be hooked up to this widget which is this button this radio button widget that allows you to select between various strings okay and the same thing for the other one we say this window argument is actually I mean they're the same text but they're here this is an argument here that's referring to this thing in the global namespace which is this insulator so we're connecting this radio button in this insider to this text and this integer argument so this function here doesn't know anything about widgets but we've transformed it into a function that does know about widgets and in particular it knows about these widgets and it connects those widgets to these so unless you write your Python make some widgets and then connect them up okay and so now when you do that your widgets work you can select everything you can put it together in columns and rows just like you did before this is the same thing that you got out of here it's still a panel if you print it it's a blob it's not just a blob though it's a compositional thing one of them is a function one of the column in the column is some markdown and a couple of Fighters that's what we are seeing okay so we've just gotten rid of all the magic most of the magic instead of the inner magic of interact we're explicitly thinking about what would its mean we might want we're explicitly instantiating them and we're explicitly connecting them up to the function okay hopefully you can see that we're not relying on magic as much anymore you make all the decisions about which widget you might want and not want about what the ranges are here the ranges are declared when you declared the widget yet about it starts at one and goes to 60 you told it that you can tell whatever you want there's no it doesn't discover it there's no magic it does its thing because you told it to okay so this is a very explicit way but it's not much more code really it just requires you to think about widgets and in functions differently than the interact one the interact one said dammit make this thing interactive this is taking a step back alright I'm on a project now I'm gonna make some widgets I'm gonna connect them up it's a slightly different task but the result can be a very similar yes you have to pass the hint that tells you the type of widget you want I never do that so I don't know the syntax but it's in there somewhere the interact function takes tons and tons of hints so if you want to deeply like double down on interact and only is interact there's probably a way to force it to do the things you want in very many cases me I say uh I hit some bit wall I said now I don't know I'm just gonna make it myself whatever you want okay exercise to do the same thing you did before but now start with some widgets whatever the heck wouldn't you want and then connect them up using this dependence decorator with some function that displays something viewable you already had a function to display something people just copy it so it's some widgets in front of it and add this little dependents thing so it'll be something like this you make some widget you make a function the function has a dependency declaration and then you lay it out as a panel row or column or some combination of that okay all right have fun oh by the way how many people are running locally on your own laptops how many people are in the cloud happen an cool glad it's working out both ways the local one you can take with you the cloud ones will turn off tonight so that's we're paying for that yes yes I'm just checking to make sure we're not over time we got plenty of time okay great right here so we can do whatever we want we can have right here it's a row of this plot and some widgets so if we want the widgets on the left or both places we put them on both places it's a row of widgets plot and more widgets and if we don't want that we can have a peon dot column yeah now we can do a plot with the widgets and then a column that has a plot and some widgets there's obviously a bad design but do what you like and this is not the only way you can do row and column panel has other things you can do it you can have it you can say divide up your whole screen into grids and put something on this grid cell and this grid cell there are lots of options this is one of them we try not to overwhelm there's a responsive mode correct yeah it doesn't know how wide this widget is going to be when it's doing that yeah there you do in this case you do peon panel P&I panel is a function that turns whatever the heck you give it into something the panel can deal with it and so if I do that it won't have done anything at all but it gives me a way to now pass an argument to this which is with equals which isn't working for well because me of the row also needs it I don't know but basically you should be able to pass pass arguments to whichever one needs to be controlled whatever the reason that is not achieving my goal there but but yes you should be able to anything that can be displayed is displayed by calling this function on it this function can take arguments what that function does is like me and up panel five if you see what that is it's a string so I can instead do and pain that's true it's the same thing so basically panel delegates to some pain class if you go to the panel website you'll see a whole bunch of possible pains panel tries to detect what is this thing and allow you to pass arguments to it if you want to do have more control you figure out what that thing is then you can find out you can do the help on the end of pain and see what our arguments it has and so it has bear it has a good number of arguments and then you can mess with those yep yes you can throttle I don't know a way to throttle it's a panel level when we do throttling we do it at the underlying plotting library that is a that is a good question the author of panel happens to live in Berlin so he's not here today he was here actually a couple weeks ago he can answer that for you that's Philip rüdiger it's and if you go to panel it have a link to our git er site you can chat with us we're also setting up a discourse site that sort of question is is well address there because yes as you set up complicated apps with you'll want progress bars to appear in the right spots you want throttling panel does have progress bars throttling is an answer I don't know it was great that we have not covered that but I will answer that pretty quickly and so if you have something that's working for you in a Jupiter notebook you do dot show and it will launch a web server on that particular port this particular is a very unambitious dashboard but it works anything that you can display basically anything that is the result of PN panel or pin row or column pretty much anything you can display just a dot show to it and that's mainly for debugging because it it means that you still have your jupiter notebook running we've also launched this separate thing and then we'll get to what you want to do when it's not debugging but well when it's not debugging what you do is go to your command line and if you actually want to launch it separately you will do so I'll make the text a little bigger but it's too small for me to read to do that okay so what I did was take a backup copy of the talk that I was showing and then run a command called panel serve on that and what that did is run through this all the cells in this notebook it's a backup copy because I've been messing with the one I was showing you who knows what's in there by now but this backup copy is what I started with no actually I ran the original instead of the backup I should stop that okay so there so I took a run this command called panel serve and gave it my notebook and what it's going to do there is go through every cell in a notebook and execute it and if it finds something that can be served as a dash board and I'll show you what that means it will serve that as a dash board so it it launched something at this URL I could have run this on an AWS server and it would make a fully public thing this is just running on my laptop so it's not public but okay so it it launched this thing now what is this thing you actually have to look back in the notebook for something a little trick somewhere in my notebook I put a little hint I did took some panel object and I said it dot servable that does nothing in the notebook but what it does is if you save your notebook and your right panel serve everything that's been marked servable will now be launched as a as a dashboard so this is how when I'm showing you these things any of these apps behind this app as a notebook a notebook was saved a disk on a third server somewhere in digitalocean there's a command was run panel served the name of the notebook this thing in the notebook was mark servable and that's what got served so it allows you to easily transition between a notebook fiddling around debugging a standalone server deploying a standalone server and leave it running forever these things are running it examples top obvious comm they're just running forever for until when they go down they restart and so these are public deploy things if you visit them and you start messing with it something happens in Python and it starts consuming memory somebody else visits it eventually the machine's gonna go down this is there's a Python server behind it with a real computation and real data and real actual work to do it's kind of unlike a typical website which just bumps it all down all it has to do is deliver some bits to your browser here when you hit something things happen on a server costs are incurred and so that that's both good and bad it means it's bad because if you wanted to be Bank of America and have a website this is not a good way to do it there's a million people visit it it crashes immediately but if you're a data scientist and you want to get your work out there and you want to leave your work in Python just a great way to do it so that's who our audience is supposed to be alright so that's this is the selection called deploying panels if you do this on your own laptop this will work like it will launch this thing if you do this on the cloud server I actually haven't tried it I don't think it's going to do anything because your cloud notebook is totally firewalled in something that only shows one port and that's the one that has a notebook on it that's gonna launch some other port it succeeds it it's running they're waiting for someone to contact it but all the context to it is being blocked and firewalled so you can run it but you unless you're on your local system where you have the authority to start a web server in the end that is not gonna be firewalled then it won't do anything useful so this is the section of exploring further called deploying panels we just told it a little did it a little bit earlier and just so you know this is a little trick that's useful in a notebook that lets you use the same notebook for a notebook and for a deployment panel doesn't know anything about anything it it when it's when you call panel serve on it it may as well been a dot Python file UI panel doesn't require notebooks doesn't need to know about Jupiter there are separate things that make it work well in Jupiter but panel doesn't depend on Jupiter so you just write a straight panel take this thing paste it into a Python file use the panel serve on that so you're fully independent of Jupiter if you have security concerns about deploying Jupiter there's no Jupiter the Jupiter is there for you to play around with not for deployment yes yeah occupancy was up here somewhere we made it's up here that's just the name of it it's a you can put it you don't have to put that but it gives you a it gives you a name to the deployment you can see it right here if you can read that it's that's that string is showing up right there it's just the title of the webpage when you serve it what should you serve it as in this case you should serve it as something that is occupancy so that string shows up there if you leave it off it'll show up as bouquet or a bouquet server or something so if I always put the string because if I launched six things and they all have the same name but otherwise they're not important okay so that's so we skipped ahead to the plant to deploying we have one more section here we have a little bit of time and this part I'm not except expecting people to succeed at this is everything up to now you've been able to use whatever flooding library you want I don't know how many how many people actually try to plot in any library so I figured a few but you could you could totally do it you just have to set it up and put the data there and so on that's fine at this point when and it would have been the same for any plotting library because all it is doing it's calling a Python function and dumping out the results anything that can dump out an HTML or SVG or PNG you can it can work with it at this point let's get to the ones where you have committed to a particular library let's say you're working with plotly or you're working with bouquet these two are libraries that give you interactive plots not just static tuff of something at that point is when you get to do really cool stuff but you have to commit to a library so everything else is general work for anything at this point onward we've picked the library we've chosen to use a wrapper around bouquet called HP plot a tree plot just a quick way to think about it is if you can do Panda stop plot you can do panda at HP block and just stop plot gives you a PNG and this HP plot gives you an interactive full book a figure so let's say we've committed to that you can do it with plot late there's nothing special it just has to be we're going to do things where you now have an interactive plot where you can drag things around and you can you'll be able to tie specific actions to events that happens in these plots and so on so in this case what do we do we just wrote this one-liner actually HP plot replaces the matplotlib plot there was like seven or eight lines this is just give me a simple H bouquet based HP plot and this HP plot supports this syntax where you can take two plots and overlay them that's overlaying this background plot with these outlier dots so that's all this is doing it's just giving this data and this data where the second one is the subset of the first plot the first one and then plot the second one not as the default of a line but as scatter plots that are orange okay so we just wrote another function here and we passed that function into our previous find outliers function so we didn't need to write a different find outliers function we just wrote a different plotting function so if we do that we can do just the same thing we did before we get a column in a row and a sum panel and oh here's where I'm using that trick I could have just put that text but here I wanted to make the text wide and not say hello to my mother and be able to put a parameter on that text so okay and then make make a little app here this is actually less work than the original matplotlib it is not dumping out an image like that was this is jumping out a bundle of JavaScript and it just it's great you work with it if you look at the code for this somebody did a lot of work on some JavaScript I didn't do that work I wrote that and so it's relying on more JavaScript magic but otherwise it's the same as before yes right yes there's not always a server so if you go to the like I showed on the data maybe you weren't here yet on the day share website figure our various websites you'll immediately see HTML Javascript or whatever sometimes there's Python behind them sometimes they're not and if there's Python behind it amazing things will happen if they're not you do things and you think oh that should have an effect and it doesn't that's basically so there's some class of things that'll be exported statically and you can do more work to make that class larger but you need the Python backing if you want access to everything and so okay so here's what we did this is just showing off a tree plot this is just getting us in the door gives you an interactive thing what we really want to do is do deeply more complicated things like this in this case so here we've tied the hover we figured out what our exposition is in the hover and we've used that to update this table over here this is the sort of things you're gonna get to but you have to commit not just a panel but to a particular plotting library and how to connect up things in that plotting library so that's not about panel it's about whatever you see you're showing panel did not make the connection between this and here you made the connection you're just displaying the things make sense so you have to learn whatever library you want to use the tools it has available for doing that we're using HV plots tools use whatever you like but here we were just to give you a flavor for it what we did is create in HP plus built on Holub use hall views has this idea of streams this particular one is a stream that listens to the exposition of a pointer so that stream all by itself when it's given a source in this case this line plot all it will do is when the pointers in that plot it reports back the exposition of that pointer it's up to you to do something with it what is it doing here is updating this table we were set up you noticed this depends now we're depending on this stream instead of depending on a widget a widget is a way to get a stream of values from a user a stream stream is awaiting a stream of values from anything and in this case the stream of values comes from not from the user doing widget but a user hovering all of us doesn't care just accept streams of all sorts of types so connect steps streaming data user events periodic events whatever in this case so we're creating a tap stream really it's not tap it's I mean that's just bouquets term but all it's just showing you where your pointer is so you could tap and it's connecting it up to the table so what if what it does here is in this plot in this function it connects it to this line and it says that's the plot I'm watching in this function we're saying here's a function that's watching you here's a function that's watching that and so that's what makes this table over here depend on what happening what's happening in this plot over here none of this has to do with panel per se actually this depends on underlying library called per am it's been echoed into the panel namespace that's a that's a implementation details strictly it's not panels problem how you connect things up we just put it there as a nicety so what this does is allows us to write a few functions allows us to have one function that returns a plot and then we do just the same thing we did before we again called interact we call it and fold find outliers and we pass it in this new function H we plot to that just does a little bit more it sets up this connection to the to the tap source otherwise it's the same thing we were doing earlier and so up to here the left and the right was exactly the same things we were doing before we had some variables we had a plot it's just like this one right instead of view fun equals HP plot it will be you be funny equals a tree plot too otherwise it's the same code and then we added one more thing which is to create this table in this table just effectively filters your data frame so anything you could do on a data frame you could do in a function like this and display it over here so and we're watching the pointer XY we could watch anything we like we can watch the y coordinate the x coordinate what data is selected a you can have a selection tool here which it doesn't look like there is but there you could have a box selection tool and the thing over here can be watching whatever it is you selected or whatever and that's how we implemented the things like that's how this was implemented it was watching the X&Y location this is also hall of use based this particular example in this case I don't know if that's still running but it's watching a selection region when I select a region here it every one of these four things is watching that selection and whenever that selection changes up that plot updates this plot updates this plot you know that selection is actually a conjoint selection across all the plots so every there's a selection object and anything you do on any of them goes into that selection object you know all the plots update when that selection objects change so if I want to look at only the things in Europe I can select the European things that have a certain temperature that have a certain precipitation whatever what's happening is that there's an underlying selection object each one of these plots is watching that depending on that selection object that selection object is also depending on the blocks so this is more complicated code to set all that up but it's the same idea that you just saw all right so exercise 3 is on your own time it just says do something you've already basically used the time because other people are waiting but basically exercise 3 was just opening it anyway and then where to go from here we already covered this section on deploying panels and then you can go from there we've covered interact and widgets there's a totally different third way to use things that allows you to be completely declarative and never declare anything about widgets but have everything also be specified use that if you want and then just follow the rest of those pipelines is one particular thing to point you to obviously it's a but if you look for pipe lines at the panel website you'll see pipe lines give you a complicated multistage app where you've set some parameters it does this and you select here and you finally get to the thing you want if you want that there's a it's an easy thing to do so there's it can have a much more complicated thing if all you want is a simple pipeline of series of steps that's actually pretty easy so don't think of that as a hard thing think of that it's an easy thing everything else is hard after that all right enjoy thank you [Applause]

Channel: PyData

Views: 6,536

Rating: 4.8956523 out of 5

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, download, learn, syntax, software, python 3

Id: AXpjbJUVeb4

Length: 86min 12sec (5172 seconds)

Published: Wed Dec 18 2019