Sophia Yang- Level up your viz skills- from Matplotlib to HoloViz | PyData NYC 2022

hello everyone welcome um yeah yeah closer to the mic I have a mic right here okay I guess I just need to speak louder I'll try okay sorry yeah my talk today is level up your wrist skills from Nepal lip to all of this uh we will talk a little bit about my pal lib and then how we build from map our lab to all of us you can find my slides through this link is on my GitHub pages so let's get started most of you probably don't know me my name is Sophia my background is in psychology statistics and computer science I have been working in Anaconda for more than four years I contribute to all of us and a few other open open source python project I also have a data science machine learning book club if you're interested in joining a book club um yeah so the reason why I'm talking about all of us today is because I'm an avid user of Hollywood's Tools in my data science workflow I actually need to build a lot of plots and visualization that's why I fell in love with all of this and starting contributing to all of this so I want to share with you my love and hopefully you will get to know more about all of this um my python visualization journey is actually starting from Nepal live uh everyone I know everyone probably have heard and uh used Maple lip right it's one of the first python libraries that I have used and fell in love with uh it is actually a very powerful tool that a lot of new users probably have not figured that out but it's actually powerful you can probably reproduce any kind of static plots you see on the internet on map Outlet using some magic and some effort hopefully uh here's the anatomy of a map of little figure where you have a figure which controls all the elements you see here and then the map figure includes one or two one more axes and then you can Define all the elements in the axis and do your plotting again it's a very powerful tool you can pretty much configure every single element in your figure and it's it can also render to a lot of different formats like PDF PNG SVG and all the other formats that's why it's well loved by academics I know Tom just mentioned this in his talk mapa lab is actually used by 17 of the papers today note that the papers is not just python papers it's all the papers are archived so it's actually quite impressive um yeah so Shameless plug if you are interested in learning more about my pal lip um I recently created a course called introduction to python visualization with with an introduction to data visualization with python using map Olive and uh panda style plot and Seaborn you can click on this link to go to the page yeah so here's my slides again um yeah so why am I talking about holidays right so as I was working with data more and more in my everyday data science job I realized I started to using a lot more high level open source tools built on top of my palette and some other uh open source Library such as bokeh and I realized that I have three needs in terms of my visualization uh first of all I am quite lazy I want to write as little code as possible and if I don't need to write any code to explore my data that's even better and suddenly I want my plots to be interactive uh I know my problem can also give you interactive uh GUI and uh plots but it's not built on JavaScript it's not designed to be interactive in the browser Tom can correct me if I'm wrong uh so that's my second need that I want my plot to be interactive uh the third need I have is I need to build a dashboard when we deliver develop deliverables to stakeholders we normally need to deliver a dashboard and in this dashboard I want to include different types of plots whether it's map outlib or bokeh or some other plots I want the flexibility in my dashboard so uh and then I realized all of us can uh pretty much satisfy all my needs uh holidays is developed developed and created by three of my friends and co-workers uh Jim Bender Philip rudiger and John Luke Stevens uh just sponsored and supported by anaconda and many other uh fundings it is completely open source so uh yeah if you want to contribute to Carlos uh that would be awesome hollowess is a python realization ecosystem it's not just one package it's actually eight packages in this talk we'll mostly talk about the first two panel and HV plot panel is a tool for you to build a dashboard so when we talk about a whole of its dashboard we're actually talking about a panel dashboard HP plot is the entry point for you to build a hollowess plots um entry plot also works with all the other tools in this ecosystem for example HP plot works with datashader in the back end to do big data visualization we'll talk about that in a little in a little bit yeah so here here are all the topics I want to talk about um all of us today uh we want to start from the basic syntax we'll talk about how do you switch between multiple back-ends for your plus and then we'll talk about how to explore your data using HP plug Explorer how do we do big data visualization how do we do streaming day like visualization how to create a dashboard and then finally how to do a deployment uh yeah I know it's a lot of content uh but I'll be high level so that you can get an overview of what holloways can do and if you want to explore more and learn more you can read the documentations later so let's start from the basic syntax here's the netplot lip syntax that many people are familiar with right you have a you need to create a figure which contains a single Axis or multiple axis is and then uh you can plot your data on the axis and set different attributes of your axis note that here I'm plotting a pandas data frame we call it DF with column X and Y oftentimes when I am working with pandas data frame I am actually using uh the pandas.plot API if I use most of the time I use either pns data frame or Dash data frame both both in both scenarios you can use the Dot Plot API right it will return you a nice looking plot and the syntax looks quite easy you can Define your x-axis y-axis in order all the other attributes of the plot for entry plot um yeah if you're familiar with pandas the apply API uh you're probably find it really easy to transition to um entry plot because you can just change.plot2.hpplot and of course you need to import the needed modules like on HP plot.pandas for pandas data frame uh you can do hpcloud.dash for dash data frame it also works with uh Rapids XL Ray and many other data formats and it's all the same same syntax so the default of HP plot is actually bokeh that's why the plug you see here is a bokeh plot here's the bokeh logo where you can zoom and pan using those icons what if you want to get a Napa lip plot using a tree plot uh yes you can so again the default is bokeh but if you define if you add the Slime hvplot.extension map.lib you will return a map lip plot for you and if you want to have a bloody plot simply do it at the Slime HV plot or extension probably you will get a poly plot right so it's super super convenient you use the same same tags and get different styles of plots um yeah and then the next thing which I love a lot is x-ray play explore uh a lot of the times when I work with new data I actually don't know what I want to plot I just kind of want to see everything uh so with each replay explore you just need to write this one love code hvplot.explore parenthesis DF um and then you get this dashboard where you can Define what's the the type of plot you want to get what is the x-axis what's the y-axis what's the group by variables right so it's super convenient way for you to to to Define your plot and uh yeah and of course you can add the title labels and styles and others uh yeah this is the tool I absolutely love um don't need you do not you don't need to know how to how to code to use this tool I just want that of code and after you've found a good plot that you love with this dashboard you can call a plot on a score code to get the the code for for this plot so this code will give you the the code of the current state of the dashboard right so if you want to copy and paste this code to your YouTube notebook and use it later it will help you reproduce this plot later okay uh the next thing I want to talk about is Big Data visualization um to the left here we have pandas.plot API plotting 11 million data points of New York City Taxi data we plot the drop-off locations the latitude and longitude and it is a scatter plot notice that it is just as blob it's really hard to extract meaningful information from this plot I know some of you might think okay what if about what if I just add a alpha value to increase the transparency I've tried that it still doesn't look very good so with HP plot you can see with a few lines of code you can get a really meaningful plot really easily and really fast you can see where the where exactly in New York you get the high density of traffic data and where you get less data right um the magic of this big data visualization licing this one line of code rasterize equals true uh where each reply use data Shader to apply rasterization basically Aggregate and counts your data and putting to cancel data for each pixel of your figure and it uses number it's uh compile cell code to C code that's to to machine code that's why it's like super fast to give you the result back and of course we applied the histogram Equalization as the normalization method for this plot you can also do a log or linear method to see which whichever you think works that works the best for your figure and then you can define the color map which is the pretty color we have we have here um from dark to red and to White and then for tiles um tiles defines the underlying Earth image so that we can overly overlay your plot on top of the tiles any questions so far sorry let me drink some water yes yes exactly yeah foreign okay so there are actually multiple ways to make this plot you can also uh have the tiles defined separately and overlay your plots um the tiles figure um it's all okay and uh yeah so that's big data optimization uh the next is streaming data visualization uh where entry plot supports streams um data frames so to on the top here we have a function returning a this is just a pandas data frame of your computer memory usage uh information right so the magic is this function periodical data frame where we fading this memory data function and ask it to return a record for us every 300 milliseconds and the returning DF here is not a pan of data frame anymore it's a streams data frame but HP plot works for the streams the data frame the same way it works for the any other data frames or just do DF and here I have the columns which is the used memory memory um and we I plotted a line plot and showing 200 records of this data and his data streams data streaming data real-time data really easily okay next important thing is dashboard right um I'm going to show you how exactly to build this dashboard in a second so uh there are actually uh sorry three ways to build a panel dashboard so you can if you click on this link you can get to my blog post um which tell you the three ways I should be plot down interactive piano binding Ram those are the three main ways there are also other ways to build a dashboard uh it depends on your your needs and each of them gets more complex and handles more complex situations and uh in this blogs post I show you how to build this one dashboard with three methods so you can compare the code if you want okay so um in this talk I mostly want to talk about HP plot.interactive how to convert your pandas data processing pipeline into a panel dashboard to the left here we have a pandas data processing pipeline where we give it a name called Pipeline and it's actually returning a data frame where we use the data frame of the car manufacturer data and we substitute this data frame by the number of cylinders manufacturer of their data and we calculate the horsepower for each origin and miles per gallon and then the returning data frame looks like this this is the five first five records you can see the value of the average horsepower for each origin and each Mouse per gallon for your car to make this path line interactive uh so here are the code uh you can you can do df.interactive to make your data frame a HP plot interact an interactive HP plot data data frame and then you can define an interactive pipeline where instead of defining actual values of your um the actual values here you can just using panel widgets to manipulate the values you want to add here for example we have cylinders which is a slider you can you can choose the number of Sliders either here and the manufacturer and also the y-axis is available we're interested in to calculate average on those are grouped button groups so you can choose different values in your wages and I will show you different results so it's quite quite easy right okay let me show you a demo very quick so so if you click on this link um it goes to a GitHub page where I have all the all the files here on the main I also have a blog post here if you want to read more the main file we are looking at is a tree plot interactive file here I open it up in my Jupiter notebook yeah so this is what we said before first of all we make the data frame interactive and then we make panel widgets here I can show you how the widgets look um so the senders you can choose four eight six because we start from four ended eight and step is two so it's basically just giving us three values and then if you want to take a look at manufacturer you can choose different things in y-axis that you choose between horsepower and weight and then we combine the piplier widget to make an interactive pipeline uh where you can change different values and then to uh and then here we pipe it to a good looking table where you can sort your table um by by different columns so you can see different pages so this is just a better view for your table and then similarly we can do dot in dot entry plot as we have a we have seen earlier it's the exactly same syntax but it also works on the interactive Pipeline and interactive data frame and the resulting plot is Interactive quite interesting right s and horsepower and then finally I use the template to have my widgets in the sidebar and have my plot and table showing in the main content and then to serve this notebook we can run um panel serve panel serve will serve your Noble locally and why is it not loading that's weird sorry why is not defined I swear you just worked a little bit earlier fake demo yeah it must be somewhere like I added a y somewhere huh I am so sorry about this I swear it wasn't breaking a minute earlier okay this looks missing okay yeah I don't know what's going on okay you can round um template.show it will just uh show up as a separate page if we want to serve it we actually need to do panels serve I should return the same thing yeah I don't know it should work anyways you get the idea uh I'll make sure it's fixed I think if you if you clone the repo and run it in your own environment it should work okay so that's deployment I just talked about panel serve which is uh you can you can it doesn't it does not only work with your Jupiter notebook file but also works with your python file uh Standalone python file so um you gotta write your app in either way and the final thing I want to talk about is that panel allow you to convert to app to um a pile Die app or a typescript app super easily if you have uh went to the fabulous talk this morning about Pi script you'll want to say a little bit about how Pi Square Works in in your browser so this allows you to to run your panel dashboard in a browser without any server needed so what you need is panel convert and then convert your app to a to awesome so in so hopefully this would work we simply run panel convert and your file you can use power die worker to run power die in the in a in a separate thread if you don't have power networker just do pile die you will run powder in the main thread and then you can output the files into a separate file called docs right here and we look at docs it will generate a HTML file for you and also a Javascript file the Javascript file only show up where we have a power die worker in a separate thread that handles the the separate worker Okay so so once we push that to our document um uh we can set up GitHub pages from the docs and then and then you can just you can see the app showing up on GitHub pages as you can see um as we're loading the page this uh the GitHub page is actually uh downloading all the needed packages and executing the code in the browser it's not running on any server so this means that you can build any of your panel apps and put it up on your own web page or GitHub pages so it's super easy for you to share your dashboard with the word and then you can see you can change different things and make your dashboard Interactive um yeah so that's that uh the fun the final thing I want to share with you is the awesome panel.org where we can see some examples of the panel dashboard so if you don't want to use panel convert and do the Jupiter do the GitHub Pages you can use panel or panel awesomepanel.org where you write your own app and it will convert to a web page for you you can log in with your GitHub page and you can host your your app on this on this website it's built by a Community member um of all of this uh it's quite nice and if you go to the gallery uh you can see I have uploaded some demo here and uh you can do a lot of powerful things with with panel and with um with the power die in pi script for example uh here's an example where you can use your video stream to detect your face and do some facial detections so here you can do different things and you can do facial detection yeah and and this also works on GitHub pages yeah if you want to learn more about Halloween make sure to check out hollywoods.org which will include a more lengthy tutorial for you to go through or check out the official documentations panel.holidays.org and HP plot all of us and if you have any questions go to discourse.holidays.org the core developers will be there to help you thank you any questions [Applause] so I've spent many hours trying to program like the custom JavaScript and bokeh um so if you wanted to customize your interactions is that you can travel scripture um yeah both you if you want to have your customized JavaScript embedding panel you can do that yeah if you search panel JavaScript there should be a page of examples for you yes I've probably looked at some of the alternative Botley Dash yeah that's a great question uh let me direct you to my blog post I wrote a blog post about that uh let me see please yep so this blog post is based on the well the the panel talks given by all the core developers from different packages uh you can see here is the table of all the comparisons for four of the uh the libraries for dashboarding we'll have the dash panel voila and streamlit um so the the main difference are yeah that's your tool support your favorite product Library uh streamlined and piano supports many uh and if you use Jupiter notebooks Panna and voila supports Japan notebooks if you need to build complex apps and use Big Data piano and dash do that uh if you need a lot of simultaneous users Dash skills the best and if you want to share your app with the public streamnet provides easy deployment but now I think almost every app allow you to deploy to the browser now so it's all pretty easy deployment I think I'm going to follow you on thank you any other questions yes yes is obvious my sister does too yeah I'm quite proud of the cat picture selections I have here yeah yeah I mean I think this one's cool yes um so you can do face detection can you do like any custom intervision model that's a great question uh the the bottleneck neck is a pie torch that's a wrong um pile die yet so you need to have high script running on power dye first in order to have things running smoothly I mean pie torch is it's hard it's on pile die but pied horse depends on so many other libraries so it won't run so hopefully you will come soon oh now he works yeah I don't know why it did not work before yeah now it doesn't have any error message okay it's a working demo now sorry about that thank you bye [Applause]

Channel: PyData

Views: 132

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, learn, software, python 3, Julia, coding, learn to code, how to program, scientific programming

Id: znm2iv_3t2M

Length: 29min 55sec (1795 seconds)

Published: Tue Jan 24 2023