James Bednar - "Declarative Python-Based Apps"

all right jim the floor is yours tells us tell us more about the negative python monitoring apps all right well i can't be as polished or well prepared but uh hopefully i'll be more grubby and down in the weeds to compensate now you have questions okay go for it uh yeah there's uh feel free to break in at any moment uh i'll start out with some slides but then i'll be diving into demos in random order or whatever i think of so uh and then whenever i'm tired i'll stop so basically just breaking any at any point don't worry about it uh this is all fresh brand new material that is only sort of half formed in my mind uh so i love to hear uh things that are unclear um i consider this a relatively small and friendly group and uh before we uh make a bigger splash uh in the big world stage so um anyway i work for anaconda i've been in anaconda for uh six years and i run a more or less a research group there i used to be a professor and i pretty much reconstituted my research group there and we maintain a bunch of software with one hat on and then we deal with uh anaconda's clients with the other hand and listen to their problems and then we put our other hat and we write software and we go back and forth between clients and writing software hopefully to write software that people care about that's that's what we uh get up to and uh in the course of that work we created um uh just created a new package called lumen uh that was created by philip rudiger as part of one of the contracts with one client but then now it's being funded by another client so it's already uh got two sources of income and uh i rep i run the holoviz team the uh hullabaz team we came up with this name because we realized we generated a lot of packages and it was hard to refer to them without by listing them so we came up with this name holobiz is basically everything my group does and if you go to hullabiz.org it'll you'll see all about all of our different uh packages and this particular package is one that builds on everything we've ever done but it's a and everything we've ever done is in python but it allows you to uh kind of harness that power without using python and the way it can do that is that the entire time we've been working with python in fact the entire time i've been doing anything with python since 2003 we've been trying to do python in a declarative style not an imperative but you just when you're using the software you say what you want rather than say how to do what you want and you'll see how that what that means here and the result is it was actually pretty easy to write it to um to take our tools and make them able to be done by a simple text file that's not about python programming you're just listing what you want the software to do and as soon as you can do that then what can you do well you can make a gui and so what we've done is and we have a gui i don't have anything i can show you but uh essentially what we've done is build kind of modern um uh point-and-click uh dashboard and plotting tools like tableau or power bi or superset we built it all on a python backed uh framework and the so why would you do that you already have such things there are already tools like that but here this has come about because we keep dealing with different government agencies different academic researchers who are running large projects and different companies and all of them there are lots of people with lots of different skill sets there's some super diehard python people and those are the ones we talk to directly but right next to them there are people who play with python and do a little bit and right next to them there are people who don't know any python they use excel and right next to them they consider excel to be too weird and crazy they'll just use uh word and so on uh and so all of these people have a role to play an organization or a big research project or a big project of any sort and we're trying to empower all of them to communicate with each other to share approaches to share tools and not have these sort of chasms in between the different groups where if you reach the limit of one tool you have to throw it over to the next group and they have to completely re-implement it with a totally different tool so if you're playing around with power bi and you get some great liking dashboard but you can't do your custom plot well you give up and you ask your python guy to do it and your python guy can do it but he has to start from zero so what we want to avoid is that and we're really thinking about the situation of the large groups of people we're working with and make everybody be able to contribute everybody be able to share and everybody be able to use the power and customizability of python without always having to write a bunch of python so you write it when you need to be custom and then otherwise you just refer to it so probably that doesn't make any sense yet but i'll dive in and hopefully make it make sense so here's uh here's a simple example and before i do anything else i'll i'll actually show how this works so here's such a text file this is actually one file but i had to paste it sideways so just imagine the bottom part tape onto the to the other part so this is three six uh i don't know 30 lines something like that um and what this file does is specify this dashboard uh that his screenshot is up here and then we'll switch to the demo portion which is basically everything else um and then uh so this uh this file here is sufficient to specify this uh this dashboard and in this dashboard you'll see but you could you can download the data file you can select uh subsets of the data and it'll update the plot you can zoom and pan and so on and the way it's done is you you title it then you have to say where uh you define these three different uh things first where's your data from and then this is that's all that's a huge topic and then of your data sources which sort of tables or chunks of data do you want to pull out of all those data sources and you're going to name those in this case we named one penguins and it's pulled out of this csv file and um at this level you can now take your data and either filter it either just throw away some of you don't want or put a widget up there to filter it in this case we specified we wanted a widget to filter it by species so you'll see that there's a widget by species uh yes somebody asked is this the iris data set but with penguins and it is completely the replacement of iris with penguins because iris came from a eugenics obsessed proto-nazi and uh the penguins did not a person who put penguins together is a perfectly wonderful person so we're gonna use penguins not iris but if you're familiar with iris this is just basically the iris dataset so uh here we're specifying that we want essentially the iris or the penguins data set and then we're going to say that we want to be able to filter it by species and uh that'll set up our data sources that we can work with uh it's called targets we're probably ought to rename it but it's um that's just based on the source the source is something external and then it comes in and the target is then something you can view so a view here we put up one view here there are lots of types of views you can have tabular views this is the plot it's a plot from a program hp plot this can be anything supported by the panel library that i'll get to that that means matplotlib bokeh plotly all sorts of stuff anything just about anything and then the rest are all parameters for um first you say uh what type of plot it is uh sort of the engine of it then you say what data it's referring to in this case penguins and then the rest are all options that are passed to whatever plotting type you have in this case the plotting type accepts these arguments to make a scatter plot of this column versus this column colored by this column and then some other options so let's see it actually work and uh go from there unfortunately i usually have two monitors so i put one on my code on one and my dashboard on the other and it works really well right now they're crammed onto one uh page but that's what we've got so um here we've got uh the data set the uh description right here um and this is just the same thing you just saw and then we're all we're going to do is do lumet serve and we'll use the very nice auto reload function which i'll demonstrate and then show and all that does is make sure it make it pop on my local browser now if i were running this in the cloud i wouldn't use show because it doesn't know anything about my local browser i would have to set up a tunnel and then and or publish it on the web and then go visit it but here i'm just running it locally so i use this very handy uh show option hey jim yes uh is there any way we can make that font size a little bit bigger which one the in this this menu uh i'm i'm not sure uh i assume was that the one you wanted to see the text file yeah sorry we had a request in the chat sure um there's a tension everything on the screen and um she says everything if possible please do you have to get a bigger monitor for everything to be big so that's a impossibility but i will try to make the text bigger and in this case i think i can make the text figure here uh it's going to make though when the the reason it was small is that i'm trying to cram this entire dashboard into one corner of my screen so but the text is big now i just apologize it looks ugly if i take this same thing and i view it full screen you'll see that it looks normal this is what it's meant to look like but let's get out of the full screen and uh actually be able to uh look at it okay so hopefully that so what i did there is i typed lumen serve now lumen is built on panel we'll get to all that panels built on bokeh bokeh has a bokeh server that's a web server so what this is saying is start up a web server using this specification file for the web server and tell it to auto reload and so on and so it loads this web server and it opens my web page and then since i my browser is pointing to it it's able to show you the contents of this web page and then you have an interactive app you can hover you can select some subset of it not that you can do anything this this is the simplest possible example um just about but you can do some stuff you can select um this is pre-filtering this is filtering at the data level so it's saying before you before the plotting program sees anything filter this data and we define that here by saying filter by species and we'll add new filters in just a minute but there's currently a filter by species and so as long as you've got multiple filters then it can color by the multiple values of species then you'll color by the species here these are all species of penguin for those who are not penguin experts can you can you show us the data set initially okay yeah that's uh that's my next step in fact because i need that because i don't remember any of the other names of the column so i do remember that for the iris dataset but those years of study of that are now useless um okay so let's add a uh here this is showing the columns of this data set and um so let's add another filter um let's also make it be a widget because that's the only one i don't know how to do uh phil wrote this i did not so okay so now let's add the widget for sex that doesn't sound right at all but uh okay here we go uh so there is a sex column in here which is the second to last one you can see it's male female and sometimes it's missing when it's missing it shows up as nan and then here's the um here's the resulting dashboard you saw that it auto reloaded just because i saved it it'll uh it's sitting there watching and it relaunches every time i save a file so now if i separate out you can look at the mail penguins location versus the female and it looks like for their build depths and their bill length it looks like the uh females are smaller than the males so that's common to many species but certain appears to be true of um uh penguins so we can use that to make this scientific discovery that male penguins have bigger bills than female penguins discovery happened right there i've never in fact done that so what else can we do here uh we can take something that per year that's probably not relevant that doesn't matter the penguin so let's take a floating point value um so all i'm doing here is i'm editing a text file in a text editor my text editor of choice of course is emacs but you would just use any whatever your favorite editor is and um okay so it looks like i now have a filter by body mass and it's figured out that the body mass range goes from 2700 grams i guess uh yeah it's in grams so 2700 grams that doesn't sound like a lot but okay um penguins are small um and then you can filter by monuments that's just because i decided to be able to filter my body mass i declared i would like to be able to filter by body mass and then lumen does the rest of the work to create a panel app if you look at the panel website panel.holovis.org you'll see lots of examples these are all apps that look like what you're seeing but they're written in python a human being thought i need to write an app about glaciers and they wrote an app about glaciers in python whereas what with what i'm showing you here a human being thought i need to write a text file and they wrote a text file and now they have an app about penguins so it's using all the same tools it's building on the same hollow biz stack as everything else we do um but now we are able to do it from a in a fully declarative way but in our group actually we have this uh this policy called shortcuts but not dead ends whenever we build an easy way to do something we also want to make sure that is not a dead end in this case what if this functionality that's describable by this test text file doesn't do what you want so normally the answer is you buy a different piece of software the answer here is you can create any of these things yourself you can write your own filter it's just a little python function a certain type of class you can make your own source you can make your own view it doesn't matter all of these things that we're seeing here are are um replaceable with your custom item and you just put it in the same directory and you put in a python file and then all of that i wanted to put together an example that i did not but just trust me everything here behind it is just python it's just write your own thing if it doesn't do what you want i don't care and that that's how it maps onto a big research group or a big group that's doing something in a company which is you've got your super high-powered python data scientist they write a new analysis thing a new transform they put it in there it does um temperature-based anomaly detection great you just say here temperature-based anomaly detection and it'll use that component and you're the person next to you can use your code without them knowing python and so you can just focus on always doing something novel and new and not redoing the same thing all the time can it go the other way around meaning take the text file and create the panel code for you like i demanded that of philip today and he says yes you can and it's at the end of the talk where i mentioned how to do that but i said that sounds too hard we need to make that better but yes exactly we should take this i should be able to take this and then go into a notebook and then use bits of it here add this bit here and add completely custom alongside this and you can but it's not documented and it wasn't part of the original design in fact the original design was insanely different from this our whole point was we're dealing with clients who've developed dozens of panel apps because it's super easy to write a python panel app and now they have to figure out how do we monitor them and what we wanted to be able to do is write a simple text file where you could each panel app is actually a rest api nowadays and so you can actually ping the rest api hey how are you doing has anybody used you recently do you are you is your health status okay are you having any errors and so we wanted to make a monitoring app that could monitor all of our panel apps or all of the customers panel apps and so that's why we write wrote this thing in the first place but you'll notice the title of my talk uh you guys said it was monitoring apps i took that word out because actually it does all apps because all apps are monitoring in some way they're either monitoring the contents of a data file or monitoring a server or monitoring something that's why you make an app is because you're trying to reveal something to a human being so i have more questions i'll save them to the end this is great yeah as you wish all right so let me go back to the slide to see what i was supposed to say oh yeah i actually wanted to show a little bit about how it how it does its magic so that you can tell now it's concrete right you can see you write a text file you get an app now um uh what is the universe of things that can do right now without having to write custom files uh custom components well just think of these three different phases stages you need data to come in we support intake that's another anaconda project intake is a data catalog and a data access solution that basically says i don't care how the data was just give me a pandas data frame and it can handle appendage data frame on s3 it can handle a painter's data frame on http appendicity frame locally as csv locally as par k it doesn't matter all these things end up as a pandas data frame as by the time it leaves intake and we get appendix data frames so we don't have to worry about any of that so lumen doesn't know about all these data sources it doesn't care it just knows you can write an intake specification which is also yaml and you do that in the same yaml file it passes that to intake and it gets append this data frame or a desk data frame back either one it does it works with both and the only thing that we've done especially is support for rest apis because there's a very dynamic intake was meant around to wrap around data files we would like to kind of fold this into intake and never have to think about it again and have separate the concerns but for the moment intake handles big collections of data files and separately we handle live uh sort of query of a server so parts are good okay so now you've got a pandas and desk data frame now you can go to town on it and then you can either uh just use it in which case the fill you don't have any filter or transform steps but very often you'll want to uh very often the data you started with is not the data as you want to see it and you just have here's where you do your magic uh here you can specify all sorts of filters like you can uh in text you can specify many of the common arithmetic operations like take this column subtract it from this column normalized by this column all sorts of things like that and that's just a a bucket of uh operations that will only grow that's there's lots of things there they're super easy to write they're very simple and very stupid they just do something and uh anytime you've come up against that something isn't there you just throw it in there and now it's there and then but then transforms could be anything like literally transform could be machine learning based or it could be send it out to people and wait till they send you it back by email whatever it can be any trend literally you have all of python at your disposal you just put a little python code in that directory and it does any transform you could possibly imagine so far so good now you've got something you're ready to look at or show or reveal all right and then you have a panel to actually view it now uh panel is also from anaconda but panel is not uh anaconda created uh the bokeh library for biz but panel is not in any way restricted to bokeh panel uses some technologies from bokeh but panel will very happily display i don't even know what this one is it'll display plotly matplotlib altair all abuse that's our own project uh um bokeh and whatever that is and uh perspective uh that's a nice little way to where the plot is not a dead end you can really just show all your whole data set and play it around in the browser you see which columns you want to show uh so perspective is like just give me the dependence data frame in the browser and i'll choose what i want to see at the final end so that's pretty much like abandoning the the whole job of writing a dashboard uh which may be what you want to do okay and then you've got either a table or a collection of visualizations uh and that's and in a bunch of widgets so you have a dashboard okay so um when we're designing this thing uh in fact philip and i just go back and forth and the other members of the holovis team um every each of us tries to out generalize the other and we say okay we can solve this problem and somebody else says oh yeah we can solve that problem much bigger three times as big with only a tiny bit more work and we go and then this is what we ended up with um uh for one thing we can easily make uh distributed dash data frames just work out of the box that means um it it doesn't take any extra work to do distributed computing with this system because of uh just because of what we're building on we've always been using das which is another anaconda project originally now it's co-owned by about five different uh organizations uh it does include uh security um basic uh security um which we initially did not put in we just said oh you'll put it behind some firewall but enough people whined about it that now you can make an authenticated app and then uh the key thing is that everything is uh is extensible so um the the rest let's just show you well any questions let me stop now uh and then uh if they're when we're done with questions or if there aren't any let's just show more complex examples as we get into them yeah i'll ask my questions now uh so um you james already know this i've been bothering his team for quite a lot for doing all sorts of reproducible like papers that are generated or presentations that are static they have the capabilities of doing so can lumen good static thing and not the web api can you just ask to save me file this html and that's it in principle yes because everything that's built on does can do that but that's part of the same question as uh going back to python and working with the rest of that which is to get things out of it that it's not a standalone python but what it really wants to do is make a full featured standalone python backed web server that's what it's all designed to do and what we need to do is unpack that a bit and make it easier to pull out components and then use them in other cases or to pull something out of its template and generate just raw html and spit that out and the answer is yes that it can do it in certain cases but we have not studied it we have not optimized it and we have not documented it so the currently not but you might explore it in the future that's a good answer but you have to keep basic capabilities if this will be needed in the future we then we we have a great starting point which is that you have a declarative specification that immediately has no code in it there is no python so in principle you could translate that all down into javascript there's no all we have to do is more work right now we have a bunch of python code that translates some things all the way down into javascript and other things call back to python to get their work done you'll see some of those in a minute but as long as you aren't using those things that call back to python you don't need python anymore it's all javascript html css and so uh we could identify certain pathways where we say oh well we've been we've been lazy and we call back to python and then we go back there we don't have to we could just go ahead and write that thing in javascript and it would never need python for that component and this component and there are certain chunks that we could identify and make the final result be completely weaned off of python so you can easily just send it by email or just put it on any web server we haven't done that work so what i see when you show me this i'm saying up until now you were able in the future before you were created before all of this people had to write all those interactive uh javascript pages they have to have web development with all sorts of capabilities in javascript and lots of libraries and not not even all web developers can do it after you translated it to python with the panel and all of this and all your apps people can now generate those using only python but now you're getting it to the level that you actually can write some text file in some funny simple language that will generate those things for you that will be static and you can send by email and then like disconnect from the internet it'll still work interactively this is what i'm saying that's the new capabilities of the book this is this is great yeah that's true okay yeah we'll have lots of things unpacked about things that just like i said we haven't realized just how powerful this is as we've gone along and we keep building it and that is one way that we'll have to retool it i think there's business opportunities galore here but more than this i'll be bothering you quite a bit excellent all right well let me actually show you the that was the simplest example of what it does let's actually show some more powerful examples uh actually before we hop to the two more examples um we do have a question from the chat uh so cg would like to know if there is django integration uh panel is well set up for django integration and i have no idea if anyone has tested that with lumen uh if we have if it does not work right now it can work it is not going to be very hard because everything is built on is uh we were paid by a government agency to make it work with django so uh we we do have dango support um for panel and we might just need to do some work for uh lumen and i haven't i just haven't thought about it excellent thank you so much for that answer okay all right let's look at another um another dashboard this one is a little longer it is uh 75 lines long and first let's just show the dashboard that was full screen uh okay so this particular dashboard is showing new york city taxi data in fact some of it's off the screen at the moment um tell me uh so you can make the apps fully responsive but not we haven't done the work to really think about what all of the um and so it sort of adapts fully to each uh size of web browser and device and so we have a good template that does a pretty good job out of the box and then we need to do more work about specifying minimum sizes maximum sizes so don't complain about that we know we need to do that so that's why things don't just immediately work beautifully but in this case i reduced the um the magnification a little so that everything fits on the screen um it already has things like this where you can shrink off things parts of the user interface that you're not using in this case i wouldn't need that so all right so what this is showing is uh 10 million trips of new york city taxi data um and um this on the left is um pickup data on the right is drop-off data this particular app looks like it is being extremely slow what it's doing is updating about a million times at the moment um when it should have very quickly updated hopefully it'll wake up it's been sitting here running on my machine and my guess is that it is uh slowly coming back into existence anyway what it's showing here is uh pick up data on the left and drop update on the right darker colors indicate uh more pickups or drop-offs in a certain area i'm just gonna reload the whole page doesn't take that long too well it's actually kind of fun here actually no i'm gonna use this because this is showing you that it's rasterizing the data this is using a program called data shader another holovis tool from anaconda and that's um and that tool takes data that might be large collections of many row point data and it rasterizes it by counting essentially doing a 2d histogram of the viewport so every um every pixel you can see it counts how many data points fall into that pixel and then it colors that data that pixel dark blue if there are lots of data points and light blue if there are fewer data points and as you can see here i actually zoomed in um and it's uh it took a long time to catch up here i have no idea why but what it's doing now is now it's got one now each pixel is actually the size of my viewport so you can see the full set of data available and then this one is not even running over here so ignore that one but this uh now that you have the data here uh the fact that there are 10 million data points don't matter because it's been um rendered down into essentially like a png so that's why we can display it in a web browser even though it's uh many more data points than you ought to buy right be able to display in the web browser all right and so here now we can automatically uh select uh parts of the uh we can filter by the various uh data sets against each other sorry various views against each other you'll see down here there are lots of histograms this whole thing is not working so i mean just uh get out of here now it is being very slow i'm guessing it's uh spending all my time processing meat.google.com instead of my app um uh in any case here i selected this histogram and what it did here is filter all of out of all of the 10 million data points it's only finding the ones for that were for high fair amounts and so on so we we have interactive cross-linking which is currently extraordinarily slow between all of these data points so that's pretty much all i wanted to show that the app does and let's actually show how it works is the map like openstreetmap or what's using for the uh well let's see it says it in the file so i believe it's openstreetmap so um no it's wikipedia which might be derived from the openstreetmap uh datasets okay you guys think that it is okay so let's just go through the uh the code and see what's in this uh what we're seeing here um all right so we titled this taxi trips and that's why we it shows up here and we say where are we getting our data it's from intake like mostly that's where we get our data and the data is in parquet format it's obtained from this url and uh it's cached locally because this is a pretty long data set it would take minutes to download um so uh until we tell it explicitly cache that thing all right and then uh we define a couple of um uh targets here we're trying to say that um we've got uh actually now the uh here we're defining views so the view is uh we're gonna have a view that is the pickup locations uh we're missing a title here it's i don't remember the syntax were titled but you'll notice that it doesn't say drop off sources pick up so probably we ought to be declaring that but i don't remember the syntax but here we have a pickup x and it says that in the background we're gonna have some tiles that are wikipedia in this case but i can change that to uh other types if i remember their names um and then here's where it calls data shader it's saying to rasterize it data shader is rasterizing the data if i were to change that to false you saw how slow that was for whatever reason it would be about a million times slower it would probably just crash the browser so i'm not going to do that um and then i'm turning off color bars and i'm defining what's called a selection group that means that this plot is going to be part of a notion of selecting that's called taxi and i declare that for each one of these this one's for taxi group this one's taxing this one's taxi and that way when i select on any one of the plots it selects on all of them this is called um linked selections or linked brushings and it allows you to find a feature in one one view of your data and see it in all of the others uh see so if this were working i would be selecting the airport and i'll say oh of the of the airport what's the distribution of bears when you start at the airport or where do you go having started the airport where do you end up going or how many passengers you have and so you can filter by that and then say oh though of the ones that started the airport for report and have a very low fare where are they going and so on so if we were working well i would show that uh okay and see the drop-off is exactly the same and then uh there's a kind of histogram it says and there's instagram by trip distance uh i decided on well philip decided sorry i don't want to take credit for his work uh bill wrote this one uh in fact he wrote all of those except for my own modifications for the simple one this is really his project but he's in germany so he's not going to be presenting in austin python meetup busy all right so the so i said uh hb plot histogram and then another hp cloud histogram and another one and at the end some options oh here's the uh here's the reason it has this layout the layout is saying that normally it'd just be a column of everything or a row of everything um here it can be the first two items followed by the next three items so you can specify how whatever layout you want and individually to specify their own heights so you get something like this and so here we're declaratively turning on the power of data shader by saying rasterize equal true now that involves a ton of other libraries and all this processing code and all this uh all these packages that ching helps uh make sure are in the right place at the right time all these different library python libraries get invoked just because i turned in uh turned restaurants equal true uh but it's declarative but i'm just saying i would like this thing to be restaurants and it's doing the the work necessary to rasterize it any questions on this um uh a particular example yeah the layout thing it's like it's numbered according to each element that you put in like the first one is zero the other one is one and the right yeah they're in order of this list and there's a feature request to be able to put a name on each one and then refer to it by name that would yeah better sorry i filed that feature request it has not been dealt with yet no no no no i i got it this is great it's much better than it was before since in family you actually had to do rows and and columns before that's right and here that's that takes code to write row of blank and blank in a column of blank and blank here that's code here it's just a dictionary or a nested list all right so uh all right that's uh taxi what's the next one um okay the next one is uh bikes these are all by the way these are all uh examples on the lumen website home page so there's no special stuff i'm doing here i'm just taking you through what what's on the home page uh but uh the it's not trivial the only thing that's not trivial about it is the environment as as again jing knows when you invoke rasterize you have to have all the stuff installed that rasterize requires and here you have to have geoviews installed because geoviews is taking all of these uh locations in london and projecting them onto the they're stored in lat long uh when they're being plotted they have to be plotted in web mercator format you have to do a projection gov handles all of that so if you don't have geo views installed you try to run this app and they'll say hey i don't know how to do a projection and so that's going to be something we're struggling with which is how to nicely capture the fact that your simple declarative text file depends on um things to be in the environment and we can easily just put in condo environment in the text file and say hey this is our environment but it's not like we want to necessarily say we'll only run in that environment so we're struggling with that and trying to figure out what to do because our apps do depend on having things available in the environment maybe it could just be a little speck saying in the text file these things should be an environment and it'll quit at the beginning if they're not but you said before that you can only define each element so you can say the element itself contains the environment no uh none of the elements involved here include an environment we could make one but yeah yeah so that's the idea just define the new elements that would hurt my head but maybe i'll think about it uh all right so first let's see what this uh what this app does um it looks like i must have hit uh refresh accidentally but that's fine um uh okay so this is uh again the um little hamburger menu you can turn this off it's probably ought to start off because it doesn't actually do anything useful there's no widgets and in this case it'll um these are all locations you can get a public bike in london so there's a data set a csv file it's got this structure these are all the places you can get a bike and at this moment that were the bikes that was the number of bikes that were there and so again we have histograms um so in this case i don't think it's showing you anything terribly new except for the table so let's focus on that um so the bikes camel um i'm just going to stop if i see anything highly unusual something i haven't discussed oh here we're declining we're declaring what should be in this hover column so when we hover over something if i try to stop there and make my mouse not move uh we're gonna add uh one thing uh we're gonna add uh the common name to the to the hover column and the common name turns out to be like the name of that area or the stop or the this little chunk of bikes is called this and uh st john's wood church is the example in this case and so you already have the latitude and longitude just to be able to plot that point and then um i'm gonna guess that it's colored by count actually i should check uh is is it colored by count hello yeah it's colored by count okay so that's why we only had to add one thing to the hover because a point automatically knows the latin long and we told it to color by the count so those are automatically in the hover we don't have to tell lumen that but we we explicitly said also add the name of that stop that make sense yes okay so uh so that's that's new um then again we're using the selection group and so if i see that i know that i should be able to uh do this um filtering so if i filter to that it should uh update uh here it's saying if i filter the number of total docs uh that has pulled out a subset of the uh available bytes it doesn't look like it filtered the map did it oh yeah it did okay i just took too big a chunk so here uh where are the show me all the places that have um uh let's say um 50 bikes where can i find 50 bikes oh okay um apparently you can find them in stratford and city of london all right i don't know what's special about stratford but there are apparently very large bike stands there and so on so you can ask ask a question like that and in the final app answer it without having to go back even to the text file let alone python programming to try to answer a question like that of course given the talk before you have to really think about the quality of your data like how did a did they systematically under represent the uh the large pipe counting areas well we we just get to the data we can't tell you if it means anything all right um okay the table so here's where we're specifying the table and so we're trying to show to set up this table right here and the table is showing the common name and the number of bytes and we told it to suppress the latitude and longitude uh so that it's only showing the name and the bytes and then the number of bytes okay and then uh we have a question that's come up in the chat here sure so patrick says that they were wondering whether you can say a few words around debugging uh error reporting from there yeah let's cause an error that's a good idea um this probably will not do anything let's see um maternal server that's not so good um what about uh [Music] so uh there's some nice um they're nice uh cases where you get a beautiful little box on your um on your uh screen here that shows a trace back now i don't know what the conditions are i i have i requested that and and it has appeared there i actually don't know how to um invoke that but as you can see it's pretty easy to recover if you've done something bad um like i said that will uh that will screw it oh there you go um i think the first thing i did which was just not valid yaml that we needed check for just completely totally not valid yaml uh because that looks like it errors before the before this bit that's capturing the error here is saying that the height must be an integer and 500 times i cannot be evaluated into an integer and sewing it's just telling you what happened here so here there's some nice uh reassuring meaningful statement of what went wrong whereas when i put just the total uh nonsense that was not a ammo spec then we got nothing useful so that's definitely something that we're just gonna have to encounter over time the more trouble we have the more we'd say oh you know that's not a very helpful message and so of course there are messages on the um terminal as well mostly they're warnings um and many of them are pointless like this is just some uh sloppiness that we had somewhere um here we're setting the number of columns of three somewhere in the code and it's being ignored because it's sitting in place where it won't be paid nobody will pay attention to it we need to find that wherever it is yeah so for the people that don't want to deal with python at all it might be helpful to actually come up with just like a simple simpler message instead of showing the whole that's a good point the uh the error message uh can like in this case let me say i think just the exception itself is efficient parameter height must be an integer you'd like to here this is showing you the python traceback that's not the context you need you need the yaml context you'd like to see that in reference to that that line of that yaml file in the context of your yama file uh so that that would be a great feature request i'm going to forget by the end of the um into the tar but i totally agree when it's being displayed in this form here it ought to match the uh the actual thing that's under control of the user which is yaml not python flow and file an issue they respond to those i know all right i'll do it an issue because i totally will forget that one that's a great idea um okay and then the last uh okay just two more uh there's two more concepts one is uh faceting uh let me try to explain fastening and uh i have not actually done it so bear with me uh so this is a this is another one uh uh here it's got a description i think that description i'll be showing up here somewhere but it doesn't um but anyway this is a bunch of data from various uh various different models run run under various conditions about assumptions about what i assume are carbon dioxide emissions uh well just pollutant emissions in general which might be carbon dioxide and these are three scenarios ranging from um not very much emitted to a lot emitted and something in the middle so basically but then we have three different models for how the earth might respond to those different emissions and so how do we end up with three different plots here what you should see if this is fastening what you see is that there's nothing in the ammo file that says model pcm in car or model myrock 3.2 or model e cham5 instead what you used to see is facet on model so i'm i'm just assuming that because i have not even looked at this file let's see that's it there we go that's where you should see that it is faceting by the model um and we probably ought to oh i can't pull the uh i can't show you the data because this is actually a bunch of data if you look at this uh intake lets you treat a collection of data as if it's part of a unified data set which is one of its best features i love it and you can say if you have a bunch of csv file files in this case they vary by they have these things encoded in the file name about which model they are and which of these different emissions they are and so each one of these plots here that's a different csv file but you're able to refer to it as a big block and then treat that as a column in the data in this case the model is now a column in the data which was just encoded in the file name and so here we're saying um we're going to fast it by file name essentially by this chunk of the file name and each one each every time you have a different file with a different chunk in this spot between precip and csv you're going to get a different plot here in your dashboard i hope that makes sense because it's super important and in fact is why we designed this entire system so that you could write a text file that doesn't know what it is you're going to be displaying and can query some other server or some other file so that's the whole point of our declarative uh um approach was that we were trying to make sure that we could write an app that makes a dashboard when you don't know what it is you're displaying because one of the things you need to know is what is there to display such as if you're querying what are all of the running apps on my system you don't know how many there are you don't know what their names are you just know how to find them you have some way to find them some server some program you run that maybe runs ps and checks all the ones that have that and then finds all their process ids and then whatever you just have something that you're going to run that's going to tell you and you don't you can't lay it out as item one two and three you just have to be able to say facet on that value whatever it is give me a small multiple with each one of these the same plot that i'm going to describe here but now for each one and so you'll look at the origin the specification this is going to tell you that it's got a hb plot line plot of time versus precipitation the model and emissions and then you're going to facet that whole thing by model so that each one is only for one of the models that probably won't make too much sense but hopefully the idea makes sense you need to make dashboards to show a lot of things that you can't enumerate except at runtime can you do very similar things that you're doing in all of yous i know that's related to panel but you can like put a plot on top of the other and stuff like that like you have those operations of or multiplication and plus that you can actually put things on top of each other yes that's an overlay and um here it's kind of an overlay by default which is these three things are overlaid on top of each other but you can also explicitly overlay and i do not know the syntax for that um in some for the taxi this is an overlay as well but that was an implicit overlay you just said tiles and it knows put tiles in the background and to explain to explain to everyone else what overlay means because you answered my question but it's important that everyone knows that you're capable of it so uh a plot like this this is just a set of map tiles this is a like a google maps and it's basically taking images about 256 by 256 or so chunks and saying wherever you are in the world give me the chunks that overlap this viewport and then show me that plot and then you overlay some actual data on top and that's what we have over here this is the data on top that's being overlaid on the maps and so each time you can see i've zoomed in into the map that's independent of the data that's there and each time you'll see the data the map updated independently of the plot updating and that's because this is an overlay of data on top of map you can overlay any number of things here it's doing it implicitly i just don't i have not learned yet the uh the mechanism for overlaying anything else think about this stack of transparencies on top of each other each one is an overlay that's right and the and the uh the counterpart to it is a layout this is a layout this is what's called an in-dimensional layout where there's a axis here and the axis is that you're the model this is the model axis going top to bottom this is a layout an in-dimensional layout in here is one there's a one-dimensional layout going from top to bottom by value of model so this is a layout and then with each one it's an overlay but um and by default you get a layout and you'll have to do something special to make it an overlay okay almost there and i think we've built up to the thing that we were trying to do in the first place which unfortunately is not running i tried to start it and i didn't have permissions what this thing is doing is if you go to our examples.pybiz.org this is all sorts of examples that we've made with our tools in fact none of these are on there yet they all ought to be on there in fact that's another to do item but here it's got a bunch of our examples and if you uh were to visit any one of those examples you could view a running version of it and it it'll show you a live demo this is a a running app that's running in the web on one of our servers in the cloud somewhere and we've got something like 40 of these running this is one this one is running very well look it works um but there are a bunch of them and what how do we know if any of them are going down any moment is there a problem with some so that that's in fact uh we told our client who wanted to model all of everything we said well what we're going to do is we're going to solve our own problem and then we'll see how much of it maps onto you our own problem is that we've got all these dashboards we're trying to monitor all of them and see which ones are popular which ones are running out of memory which ones have we over provisioned which ones has nobody ever visited and we should just kill and so uh that's this um this dashboard is actually uh monitoring um like square limit that's one of these things in here that's one of the um apps that we're monitoring um square limit it's right there so uh in this monitor app this is faceted by deployment so square limited is a deployment you just saw the attractors there are lots of their 40 some odd deployments so this is a faceted plot by deployment and for each deployment i want you to show you the memory it's currently using how long it's been up in fact we restarted all of our deployments in the past day because we our certificate runs out every three months and it ran out today so we restarted everything so everything's been up for nine hours this one this has been up for nine hours and it's restarted not at all it has um we don't have any visitor data for it yet um i think that things are being collected at certain intervals and we don't have the data for that yet so it's not showing what it is showing that it has not restarted it's showing its recent history over the past uh well since it's been up looks like nine hours showing a used memory for a while and then it's gone to active zero nobody's visited this thing recently yeah and then you can look around and then whatever down here uh i don't even know what that is it's like lightning talk i don't know why that's deployed but it's got an error in it because it's been restarted 76 times since the past nine hours that's just some junk we left that around to the server i we detected that it's there by this screenshot if it were live i could go kill it but in fact you can make it you can add stuff in here and you can make a button and you put a python thing and you said kill it i can make it be an active dashboard that does stuff um but at the moment it's just a screenshot so i can't do that and uh and so on so uh and then we have information like this is actually queried from the panel app itself that's running there so the panel app has a rest api we ask it how long do people spend at this app on average and uh how long does it take since when you first get a visitor to how long you return the results because we care about that that's how long it took for them to see the dashboard appear so we want to optimize we want time to render to go down session duration to go up that's engagement visitors to go up and so we monitor this dashboard and think about our problems and we want anybody to be able to do the same thing based on a pretty simple uh dashboard that tells them um a simple text file where they specify what they care about here we're specifying we want an indicator that has a dial that tells me the memory percentage we want to have um a dial with the cpu percentage we want to have um uh a well that's another dot um i don't know that is um okay oh sorry that was a different section those are not the dials you're seeing here there's another page just a little multi-page app by the way um we're looking at the details page there's also two other pages that are totally different that have the the deployment overview is just a table of all the deployments where you can sort by which ones are using the most memory which ones are using the least and so on in fact that one's probably uh uh so uh the other tabs um are uh don't require the same permissions so i can actually run those but anyway the other tabs are having totally different data and i was just showing you there i was showing you this one that has a bunch of um dials and then there's also this which is all everything sorted by uptime and you'll see they're pretty much all the same uptime which one's using the most memory and so on so uh that's done in this uh table this table says the uh membrane percent cpu percent and so on so i get this table because there was this chunk in the um respect and then i get to the details page and that's what i was showing you with this um screenshot which is i need a memory percent dial i need an um restart indicator which is a number i need a session info which is a number and a time series lot of cpu usage over time standard okay you can pretty much if you look at one of these apps you can think oh what would the what would the yaml file look like oh it's going to have memory it's going to say memory and it's going to say timestamp on the bottom and so on then you can look there but of course that's my last point which is that you have to know that and think about that and know the structure of this file but this is all something that can be enumerated we know what all the values for these things can be you know what's been defined and what hasn't been and so why don't we have an app for that so that's what we've got now we we can just make an app we have we know what goes on all those widgets we know what the about once you've chosen hp plot we know what all of the other options are we know all of it and so uh because it's all declarative all the way down we have this whole declarative system and so i don't have that to display only philip can display that um to demo that um but there's the whole lumen builder thing where again it's pluggable you can have different ways of designing your um your view your plot one of those is perspective which is already out of the box able to design a plot and let you fix which columns you want to see and whether you want scatter or a line or whatever and having done that then you can link them all up and lay them out so i won't show that but that's where it's headed and that reaches yet one more step away from python to people who are comfortable to interface also it's a great way to learn because as you've seen i don't know what all of the possible things are here i like to start with a big file because then i can at least delete um because i don't know what all the possible options are but i can actually guess by looking at the examples there are only four examples on the lumen website so you're not going to be able to guess all of them so go back to the lumen builder thing and you're not going to show it you tell it but is it like a a a an app where you can actually place the things on the screen and play around with them and then let's create a room and file for you um well that's uh was consists of these these components you first select a data set and that's basically a gui for intake and intake has a gui and so uh right now with one of our clients they're they're not using intake they're using different system and so we've done that's pluggable and so we're using their system we haven't really built up the intake system the intake system is also uh is already pretty solid so we can select that and then we need to be able to select the tables from uh each data set that's something perspective provides then we set up filters and transforms that's just a set of uh can functionality you select from what's available using a panel widget and then you declare your visualizations again in perspective or you could we could do a different job at the moment we're using perspective and then there's step five which is where you would like to be able to um drag things around drag and drop and at the moment no at the moment you can choose between some uh some canned uh layouts but we do have uh templates for panel apps that give you drag and drop layouts that just where the final panel app is a drag and drop layout where you see the panes here you just drag them around and rearrange them if you like and you can query what is their state and so we can easily imagine putting that in there so that you you just dump it out and give us an ugly one you let people rearrange it and then you memorize where they were we're not working on that we're uh we're in low ambition for step five at the moment but what you can do is just come up with anything ugly and then spit it out to step six and then you've got a yaml file and then you just do whatever you want to fix whatever it's gonna be limited it's never gonna be exactly what you wanted but if it isn't just edit the yaml file it takes you two more minutes because at that point it's already all in there you're not to imagine oh what is the syntax for it it's like it's there yeah well that's our idea behind libman builder because the way you show all of those steps is something that you said at the beginning you said that each person does something else and deals with something else in the organization like some people deal with data some people do the dashboard like the spreadsheet thingy and some people write the python so basically you're separating by ability to do the scam of ling and each one the person doing something different you eventually are able to integrate an entire team that's the idea in fact working with clients who are doing exactly that there are people who've never used python they're designing lumen dashboards they're hitting a wall and they say hey data science team how do i do this the data science team is writing a custom module giving it back to them and they say oh okay and they don't even know how this works they just know it works they just know that they had this need and somebody who knows how to use python it just seems like a god to them wrote some code but to them they have this menu of options and they have an out they have a steam escape they have a a way to get out of their limited menu when they need to oh my god so this will change the landscape of things that's for sure i'm trying to change the landscape all right so um what do we have to do uh we have this is a very young project the first release was maybe a month or so ago it's still uh very rough edges um number one you can't even reproduce those dashboards you see on the homepage as soon as you do you're going to find that your kind of environment doesn't have the package you need for it and we haven't solved that what we're planning to do right away is at least put up one environment that has everything in it so at least you can just use that but we don't have any mapping between yaml file environment yet or a way to do that then we need to release our dashboard builder which we have not done or it's not even committed anywhere then we need to make the docs better and add a bunch of new filters transforms and views as soon as we do that we have to make the docs better again uh it doesn't have a logo all of our logos have projects have a logo uh phil did not like the logo my daughter designed which was very rude so he brought that no uh and then we'll get to some of the things we talked about before that you've brought up jacob which is how to take uh tasks uh and users and improve the flow of everything between everybody as soon as you see something on the screen you should be able to work with it in panels as soon as you see something with hp plot you should work with it in aluminum and vice versa and that's going to take a lot of time so that's last on the roadmap it will never be done but it's definitely a goal we want to make it easy we want to remove barriers between personae and tasks and just make it easy to just focus on the thing you're trying to do right now and not have to redo all the stuff you didn't want to redo okay and then it's easy to install but what's missing here is all the other packages you also need data shader you also need to ask for one of them you need uh various other things and it's not in this list so if you want to actually get started though if you're willing to encounter some pain now uh lumin.holovis.org is up it's pretty light on documentation but it has what you need to do what's shown if you want to see this is all built on all of his tools there are lots of other olivist tools i mentioned a few of them uh if you want to pay us to make it do what you want please do but we're actually getting paid pretty well to do because we have a lot of people with a lot of goals and they are willing to say okay this is our roadmap this is what we're working on we have a big roadmap for illumina ahead of us so um it's likely that we're going to get to what you want so you pretty much don't have to pay us for this particular one it's well funded and just has a roadmap in place someday funding will drop off and we'll probably want to pay us for that you can certainly pay us for anything else if your company wants something else from holobiz please do um and then uh this is all work from philip rudiger it's his birthday actually um he's not here and he's not in austin anyway so you get stuck with me thank you for your attention all right thank you so much all right yeah that was great yeah i thank you thank you jim and uh uh thank your concept yes to integrate the uh presentations today so yeah i'm gonna stop the uh recording right now so wait wait wait wait wait before you do uh sorry we have a couple questions from sherry earlier yeah so jim you were showing the um [Music] uh the like visualization applications monitor um so i know that it was just like a screenshot of the thing um but is there any way to look at that and like see which ones were the popular plots that people were looking at uh yes um in this case we would have needed to put that data on here this particular table doesn't have the right column but in that if we at the details page if i have the authority to see what's on that page there's a sort over here uh down here and so uh you can sort by session duration for instance or uh total visitors that's this off the list here i think that's off the screen so all you do is it has to be sorted by something all of these facets and the facets are currently sorted by the memory usage increasing memory usage it looks like so you would um click on that and then select reverse and then you'd be able to look at uh the most popular ones or the most whatever awesome thank you for that i'm glad you brought up the memory sorting because that was sherry's next question um what kind of memory usage is being displayed is that cache memory external memory this is memory uh this is these are kubernetes containers and they have so this is the total memory of that uh particular container and the total usage of that container as kubernetes defines it and so there are a lot of holes in that definition this is very virtualized so um i can't be any more specific than that you'd have to dive into kubernetes but what it does tell you uh if you looked at the um at the overview these are kind of more uh real these are about the underlying hardware that the entire thing is running on and of that interlinked hardware we're using 23 well that doesn't help you tell about particular apps so you have to get these proxies that are per app but that's in a virtual environment with lots of baggage around it so it won't really tell you the bytes that are actually used on your source memory very exactly hopefully that makes sense it's about kubernetes not about lumen awesome thank you so much for that answer and uh thank you florin for holding off on pausing the recording yeah no problem yeah let me see here you go yeah yeah jacob has one for sure actually they can wait after the recording all right all right so stopping requiring right now

Channel: Austin Python Meetup

Views: 181

Rating: 5 out of 5

Keywords: python, python meetup, austin python meetup, python lumen

Id: 0oRBvnCiUJE

Length: 68min 58sec (4138 seconds)

Published: Sun Apr 18 2021