James Powell: Building Web-based Analysis & Simulation Platforms | PyData London 2017

[Music] let's get started this is building web-based analysis and simulation platforms with react flask celery bouquet and numpy we're at PI data London 2017 it is May 5th Friday May 5th 2017 I'm James Powell it's my pleasure to be here presenting this tutorial for you I don't have any prepared material for this this entire tutorial is live coded I hope that you will join me and live code enos now before I get started I have just a couple of comments I'd like to share with you just to contextualize this tutorial so you may have seen me checking tickets earlier today and just thought that with what I do but know I'm actually a volunteer for an unfocused I do quite a few things related to Numb focus and PI data but unfortunately whenever I come to these conferences then I come to a lot of these conferences there's always this one question of James what do you actually do for work and I'm so tired of that that I thought for once in my life I'll actually give a talk about what it is that I do for money so this is very much unlike any of the talks I've ever given before and that it's useful and work-related and something that you might actually be asked to do for somebody and get paid money to do it as opposed to they're usually completely frivolous stuff I like to talk about there are a couple of problems with this tutorial when I showed some of this to Ian and Kelvin they said that's impossible to give you'll never be able to give this tutorial it's not going to work and these they're probably right that said a lot of this tutorial is live coded and you are encouraged to follow along there are a couple of things that will cause you to struggle and I have one or two small solutions to help you out with that what's going to cause you a struggle is if you're a slow typer we're going to go very quickly because we only have 45 we only have 90 minutes and there's not much we can do about that if you have if you are if you have difficulty with window management you're going to have a tough time because to actually make this work we need to have five run live long-running processes running simultaneously and it's just going to get really I see because you have a lot of tabs open in your terminal so I'm sorry there's not a whole lot we can do you know I hope you have a really nice tab terminal or you use screen or T MUX the nature of these technologies that they have lots and lots of lots of moving parts now to help you out you can see there's a bitly link right here and you should go to that bitly link and what you'll see is you'll see an empty Dropbox directory this Dropbox directory is where I will do be doing all of my work so at certain points in this tutorial when you may realize that you have gotten behind on some parts you can just go into here and download exactly the code that I have it's already populated with a couple of pieces that are necessary in order to make this tutorial work I have tried to minimize the number of pieces that you won't be able to write for yourself let me give you a little bit of the context for this talk so it's this link right here and then if you go to that link then you'll see the Dropbox so I would keep what I would suggest is keep a clean web browser instance open with Dropbox open and then later on when we actually build our framework use that same one if you get confused with the other tabs that you have already open you're going to have a bad time also open up a terminal and if you can create tabs in terminal create a couple right now if you use T MUX jump into a tea mug session if you use screen jump into a screen session because we're I'll try to minimize the amount of jumping around but I simply can't if you do not have anaconda installed I would make sure that you have anaconda installed as soon as you can it is not necessary to use to have a Nakano to use these technologies however I have designed this tutorial such that you do not need anything but anaconda to get it to work so will actually install Redis celery node and a couple other things all through Conda and a couple other things too pip so as long as you have anaconda installing your system you should be okay if you don't then it's going to be kind of tricky you'll have to use whatever your native package manager is on homebrew or ports or apps or whatever that is if you're on Windows you might have difficulty getting some of these dependencies simply because you know I don't know I don't know for certain that Redis will run on Windows and unfortunately the second half of this tutorial is extraordinarily fiddly so let me give you the premise for this tutorial a lot of what I actually do is what would not be classed as data science work when I was at you know last year at PyCon there was a discussion in the core developers room about some of the type work that Guido has been working on this is pet 484 I believe and one point that I brought up to Guido was that some of this PI pending stuff really isn't that relevant to data scientists because you can't really add meaningful type insan to data frames or nd arrays and that's what you need most most code that a data scientist writes is taking a data frame pop out a data frame and the mechanisms by which they expose structure the typing mechanism storage the expose structure don't really accommodate any real checking of that that discussion evolved and a very interesting point was brought up by Guido which is essentially he sees that there are two people two types of people in the world to use Python there are the people who are paid for the code that they write and there are people who are paid for something that they write code to do most of you who are data scientists aren't paid for the code that you write your task is to optimize some business process or to decide whether to buy or to sell something and in pursuit of that you write some code this is not what I'm paid for I am almost always paid for either advice on how code should be written or code itself my deliverable whenever I actually work is code itself and so most of the projects I work on or what you might call the kind of corporate terminology is digitalization projects and older terminology that we still use sometimes is what you might call Excel remediation the idea is some analysts somewhere has written some analysis with some blacks box model and it needs to be put into production it's not enough that it runs on their computer and an eye Python notebook it actually needs to be part of some production system and the key to that production system often some kind of web presence in the case of these it's often analysis platform where you have a core set of analysts who know how to design and tweak models and you have some analysts who only know how to use models it's a business analyst not data not that a scientist or data analysts and so they need to have some interface by which they can use that model that's suitable to what they know and what they know is not Jupiter or Python or any of that more broadly you can imagine as a data scientist your ability to deploy or to send your code other people is oftentimes limited of what you can actually stick into a Jupiter notebook and Jupiter notebook just I P I PI and B files it's a very poor way to distribute anything but fairly simplistic narrative analyses if you have supplementary libraries then you either have to direct them to pip install something or you have to do some weirdness with sending them a zip file containing a requirements txt and an IP MB file if you have data files you have to zip those up and email them to somebody as well if you have parts of that code that you cannot share you're stuck for some of my projects we have simulation platforms physical simulation platforms where certain constant factors in the equations which we're optimizing over have been discovered as a result of very painstaking industry analysis they are proprietary nobody on the project is actually allowed to see the constant factors that are used in the equations we are given dummy constant factors that we use for prototyping the system and then at some point when we hand it over to the client the client then goes in and actually fills in those constant values because there's literally ten scalar values that they can't share with us because they're worth more money than the entire team put together and it's just because for this particular proprietary business process for these particular provider of business processes this is knowledge which they can't share and so you simply could not distribute these analyses as a jupiter notebook because how could you hide that proprietary information that said beyond that motivation I'm just going to assume that all of you have some interested in seeing what you can build using these technologies and I won't push that any further however it is necessary for me to mention a little bit of motivation because when we get into the second half of this tutorial you're going to see that the technologies that we're using are very fiddly and they are not nearly as easy to use and some of the technologies we'll use in the first half of this tutorial namely almost everything the JavaScript ecosystem is going to be I don't want to prejudice you because you've already been prejudiced against JavaScript by reading web blogs by hearing from your friends by following people on Twitter you're already prejudiced to think things like this the JavaScript ecosystem moves too fast nothing is stable there are no ten-year libraries you know you use something you write something today and by tomorrow it's out of date I won't say whether these are true or false what I will say is it's clear that there are many more moving pieces that are necessary to be effective in that ecosystem and what I will say is for most of you who are paid not for code that you write but for analysis that you do even though you may need to build these systems as part of some requirement that is either imposed upon you or something that you want to do you probably don't want to read any documentation I have about 30 seconds time span of attention before I get tired of reading documentation I am exceptionally lazy and I'm not very smart either and so I can only read through about 30 seconds of documentation before I get bored which is why when I started getting into these technologies as part of work I had to do I struggled because in order to pick up one of these things you have to read through and invest a lot into a set of build tools a set of execution environment a set of languages frameworks components and it just is a lot to think about so one important caveat for this tutorial I would not call myself a react a flask a celery a bouquet or a maybe half of a numpy expert but I'm not a react flask celery or bouquet expert by any measure this tutorial is designed for somebody who is not a react flask celery or bouquet expert and who doesn't want to be who wants to use just a bare minimum of these tools to accomplish a task but understands that at each step of the way you're going to be making compromises you're going to be incurring technical debt you're going to be doing things and it's a more expedient but a informal or sloppy way and that at some point in the meaty to far future when you say I need to address that technical debt or when I do something a little bit more sophisticated than what we see in this tutorial you will then invest the time to learn that frankly the problem I see is a lot of these technologies take too much upfront investment so what I'm going to try and present to you is the absolute bare minimum upfront investment to actually get something to work and in order to do that I'm going to compromise significantly on the quality of the work that we do I'm also going to compromise on the maximum possible sophistication of the example that we have the example that we have and the use of these tools will not be sophisticated so if you are already a react flask celery bouquet or numpy expert there's almost nothing in here on any one of those topics that will be interesting to you what it's interesting in this tutorial is the combination of those so without further ado let's get started and I'll give you the motivating example that we're going to work on in this tutorial so many of us when we were in school we took a probability class and in that probability class we learned a little bit about Monte Carlo simulation and so the motivating example I wanted for this and you'll see why I chose Monte Carlo simulation is some simulation that takes time that's slightly more complicated than maybe a linear regression and takes a look and has memory and processor requirements in that it's not something that you can immediately do in line you actually have to go off and create a job server and so just to review the core concepts I'm going to pull in a couple of libraries from numpy so I'll just pull them square root and linspace and you don't have to follow along in this part I'm just giving the motivation and I'll tell you when you when you should start following along although don't don't think that since you don't have to follow along from the beginning that means you can you can goof off for the rest so here we go we have square root and lid space and uniform from numpy and so that the quintessential example of Monte Carlo simulation is estimating the value of pi and the way that you do that is you draw a unit square and you have I have fourth of a circle and you drop random random values on that and you count what's the ratio of values inside the circle versus the ratio of values outside so in numpy it would look something like this let's create a linear space from zero to one with a thousand people's let's create YF which is the square of these and so this will actually represent what our unit circle looks like so here's our X and y value our unit circle what I'll do is I'll import boquete plotting figure import figure and IO from okay and I'll actually plot this for you and so this is your very simple bouquet tutorial we'll create our figure and we'll show our figure and there's one actual thing we have to do in addition to this we also have to grab in output notebook and instead of something like your magic command your matplotlib notebook in bouquet you just do output notebook as a function and it sets you up to do bouquet output in that notebook so here once I run this cell you can see I have an empty plot let me hide something so you can see as much of this as possible I have an empty plot in this plot I'll just do a line plot with my X is my Y values and I'll make it blue okay so what you can see is we're just going to drop pins on this and see an account the ratio of the ones inside and outside so what does that look like well let's decide the size of the simulation will do so we'll have a thousand points and we'll just create some random variables drawn from uniform distribution drawn from zero to one where the size is this size and two points each is XY coordinates so there we have our random points and then to calculate how many are inside we just take the x coordinates and square them and we add that to the y-coordinates squared and we compare that against the radius so now we can add into this diagram will grab the x coordinates the ones that are inside and again you don't have to follow along too closely this is just a motivating example so there's those those are the things that we've dropped inside and if we look at the ones that we dropped outside we'll see the essence of this simulation approach and we'll just negate this to get the ones outside and make them read and there you can see the ratio of these values gives us an approximation for pi so our PI approximation would be the sum of the number that are inside divided by the size and since this is 1/4 of a circle we multiply times 4 this gives us PI over 4 the ratio of these the ratio of the area of the of the inside the circle to the entire unit square and so let's pull in one more thing from ipython and now I'm just showing off here display import la tech display we got to show off a little bit because they're going to be really disappointed at the end of this tutorial display left X and then we'll just do a so we'll do some fancy lot X because we're going to approximately dot format pi y equals true oops hopefully we got a sorry this should be displayed laughing thank you so we've already we've are so there we go so our proximation is 3.08 and we run this we'll see and what we can see if we increase the number of samples it gets very slow and our approximation does not converge very well it's a pretty poor way to approximate pi because you can see with this many samples for how slow this is doesn't do a great job in general there's not a particularly good approximation approach you can see with this much data we only get two decimal points but you can you can imagine that this is now from this point kind of a black box and from this black box we'll just imagine we have some model and that model is slow and we can't run that model every time somebody requests because there's an end point request we have to find some way to run in a job and determine you know what status that job is and if it's pending or what not and so we'll see those pieces kind of inch together but this is the core of what we're going to do now what we want to do to make this more interesting as we'll generalize it what we'll try and do is we'll generalize the problem of computing the area under a curve computing an integral and so in our application we will take an arbitrary equation in terms of XS the X values and we will take the extents on the x axis and just for hinting the extents on the y axis and we'll actually approximate the area under that curve using our Monte Carlo method and that's our black box model the reason we choose as blacks model is a couple of things one there's a visualization component of seeing the dots and using the dots to kind of drive some understanding of why the approximation might be good or bad it takes a lot of time to run these simulations it may take one or two seconds so there's certain considerations for that and three we need to be able to customize we need to do something more than just running a single black box with set inputs we want to think can we do something a little bit fancier like allow the user to specify what the algorithm is that's the motivation so with that I will save this notebook and we will not see this notebook ever again so we'll say that now one thing if you are on the drop box you should see should be running live but you should see everything is getting saved well you can download them but everything is getting saved live okay what you should do next is the following in the folder where you're working on this tutorial create two folders actually create one folder call that folder back-end we're going to work on that back-end side of this we're going to take this algorithm we're going to write a Python script to implement this algorithm from that Python script we're going to wrap that in a REST API that takes JSON data as input and returns JSON data as output from that API we're going to take the algorithm and we're going to spit it off to a job server using celery that should take us about hopefully 30 minutes once we have that we'll build the web front-end using react as a single page app hopefully we can get through the next 30 minutes we have about 20 lines of code to write but actually they're not even that hard for 20 lines of code so let's get started I have a brand new Anaconda environment which I'm using for this tutorial now you can see here's my notebook running in my team accession here we've got really dark in the room and we switched from check that out look at that I try and use Python 3 because I don't know I don't know how much of this will work in Python 2 vs. 3 I don't think there's anything where it should matter if you're using Python 2 but you'll have very slight differences and getting as close to my environments as possible will make it easier for you ok so we're going to start by just writing the algorithm out for computing exactly this now in this back and folder I want to show you a couple of things that you could ignore in this Dropbox folder ignore this file n I'm going to use this in this example I don't like condo environments I don't like virtual environments so I wrote my own version of them you don't want to use it I use it because I think it's better but nobody else agrees with me so just ignore this file M it's just for my own setup and ignore when I use it just make sure that you have everything installed you know you have your anaconda on your path you can ignore the IPIN be checkpoints of course you can in order to get repository I won't be committing these to get as I go just because I don't think we have enough time to do it at that pace focus just on the backend and the front-end folders don't download the front-end folder first because it has a bunch of junk in it that we need to talk about before we can look at so just create an empty folder for back-end in this folder we will be writing three Python files and we're going to start with the very first one and for the very first one we don't actually need anything interesting installed because we're just going to write raw Python code and so our this is going to be called algorithm Pui and one thing that I'll do is you should be able to see at the bottom of my screen right here if you ever curious what flowerin just check the bottom of my screen right here and I'll tell you what following we're in Dropbox tutorial algorithm WI so we're actually in the wrong place because we need to be in the back end folder that's mistake number two I wonder how many we can take bets on how far we'll get by the end of 75 minutes according to Calvin this will crash and burn in the next five minutes so let's prove them wrong I would really encourage you to try and color along at this point everything that we'll be doing I'll try and go slowly enough if you miss any steps just save your work off to the side and you can grab it from the backend folder I'll make sure that my code works but part of this tutorial is to tell you this I gave you five technologies some of you in this room I wish I had time to survey where all of you are but I'm sure some of you in this room have never used four out of those five hopefully everyone's use numpy there's a chance that some of you have you have never used the other four and what I'd like to tell you is even though they're not easy technologies and they have a lot of depths there is a shortcut path through becoming effective at building something that can do what its close enough to a real-world problem without having to learn that much and you'll get a lot more out of struggling with following along then if you just passively you know turn on the TV sit back and watch the show okay we're going to we're going to do something very simple so we'll say from numpy and Portland space we need a linear space because actually we don't need the linear yeah we do and we'll also grab from numpy dot random will grab uniform and the reason we'll use uniform is that it gives us the low and the high values numpy dot random gives you something on a uniform interval of 0 to 1 but you can think that if we're going to be able to set the integration bounds then we're going to want to be able to have a nicer interface for saying give me a random value between negative 2 & 2 without having to multiply and subtract we'll create a function called approximate and it'll take five arguments f the function a and B which are the integration bound so you'll be integrating from A to B will take hints C and D which are the bounds on the y-axis when we generate random values remember in the case of our PI simulation we were dealing with a unit square here we're dealing with a sketch we're dealing with the unit square so it didn't really matter what we did but here we're going to want to be able to integrate over a specific SLO to start up ok takes awhile to actually repopulate the grass this is one thing that MATLAB is a little bit better at I'll let you know when that's ready but we were doing with the unit square so we didn't have to care about we were just generating random numbers on that interval of 0 to 1 on the X and 0 to 1 on the Y here since you want to be able to integrate anything arbitrarily we want to be able to know as a hint we have to know what the X values are but we need to know as a hint what the Y values are just so that we don't start generating random values that definitely would never be in any relationship to where the where the function is we want to make sure that we generate a good portion of them beneath the function and then take the ratio of the areas now one thing that makes this kind of tricky is that we're going to take this function as a string so we're going to allow the user to input a string and they're going to use the variable XS X's for what the other is that the endogenous variable the X is whatever the x axis for the X variable and in order to do that we can use the built-in Python eval but you can think we're already you know that's kind of ugly because then somebody could misuse our system as a very slightly barely safer way to do this in sin PI dot parsing that simple parser they have a eval expression so that'll make sure that at least you can't pass arbitrary Python code here you can only pass what a simple expression can include which is basically all the arithmetic operators plus some function calls now to make that work we need one more piece we need to import numpy and this piece is a little bit fiddly so what I would suggest is you might just want to go to Dropbox and get it import eval expression needs to take a dictionary for the local variables and the global variables it cannot take a module so in order to give it access to everything in numpy i'll just give it a simple this and so what I would say is this line is a little bit tricky to this line here eight-nine is a little bit tricky so if you want to do you can go to Dropbox go to backend refresh click on algorithm that py and just cut and paste these two lines that might not be a bad idea if you find it you don't want to rewrite that from scratch it's only a very small thing around the use of eval expression and then we'll make this quite easy well generate uniform random very you will generate random numbers on the dist on the interval from A to B uniformly of the size that was specified will generate Y variables from C to D of the size that was specified and then we'll just compute how many are underneath the curve because we're trying to integrate under the curve and so all that requires is that we say the YF values that are underneath eval of the expression F with my locals or I can make this I'll just do this like this this is maybe a little bit easier for you and our numpy dict and what this will allow us to do is this one single line of code that allow us to pass a function as a string and to evaluate the function using the X values that we have and to compute what's under the curve and what's not and then simply enough we just return the sum of what's under that's the count of things that are under divided by the size times the area of this box and the area of this box is simply the D minus C extent times DB minus a extent just to just the height times the width and that's it that's our Monte Carlo approximation for the area integral now there is a better approach for this there's another formulation of this that is better than what I have here but it'll make it a little bit harder for us to generate that little dot diagram so we'll go with this worse approximation so we can get the pretty pictures later on now that we have this we'll make this a Python script and this is where most Syrian alla seas will probably start they'll start your ipython notebook and then you'll want to actually make them available for people to use so we'll turn into a Python script and we'll grab F ABC and D from Arg V we'll say size is equal to int of rv6 if it's provided we'll give it related to the size optionally or you can just presume that they have to take it don't worry otherwise we'll assume a size of 100 and then we'll just do a little bit of conversion because these are taking in a string so we'll just say that's map int a b c d like that okay so just make sure those are all in so we can swap the order doesn't matter then we'll just say print integrating blah blah blah from blah blah blah - blah blah blah with blah blah blah samples F a B and the size and we'll say the approximation is equal to welcome procs ma b CD and sighs on script that does the core of our Alice's now one thing that you'll find is as part of a business process if you were to take the amount of time in the amount of money that's spent actually building a business often a minority of it spent on the model itself and a majority would spent on hiring a lot of people to put it into production and so what you'll see is this is basically all of the data signs in this talk about five lines of code and the rest of it is actually putting into production it's my experience that oftentimes when you look at the size of organizations the number of people who are data scientist versus data engineers the amount of money that's spent putting something into production versus doing the initial analysis and the amount of time that's spent doing the initial analysis versus putting into production it matches kind of the same you have five lines of code of in a real meaty numpy and then lots of infrastructure pieces so we're basically done with a numpy part now having done that I'm going to split my pain and I'll make it so that you can see that and I'm just gonna run this so I'll go into my back-end directory and I'll just say Python three algorithm and of course ignore the M that's just what I do to get my environment and I'll pass square root of four minus x squared as my function I'll integrate from zero to two I'll give it hints of zero and two and I'll let it work and of course I have to call it XS there we go so it's approximation for pi is 3.0 8bim 2,000 samples 3.1 to if I run it again one nice thing you'll see is a no focus remarkably slow at creating very large non PI nd arrays it actually is much faster in raw code because you can see if I did this in the notebook it actually is slower even if you ignore the time that takes to the graph now I don't know if I can actually do an NDA with 10 million samples look with 10 million samples the best I get is three point one four two three let me do that one more time let's just time it let's see how bad this this is not a very good approximation because you can see it's slow it converges very slowly and the best I'm getting for my approximation of pi is two decimal points after 99% CPU usage in 10 seconds but many of our black box that's another thing that I've seen production really bad models so you can instead of instead of taking this as a criticism of the Monte Carlo approach for this particular problem just take this as one more piece of the real world which is sometimes what you're putting production isn't a very good model to start with but you still have to put into production control to God is everybody ready to move on or do you need a second or two if you need a second or two I'll give you say three minutes and in those three minutes anybody who has any questions can ask them and everybody else can tune out and just copy if you don't want to copy along you can always just cut and paste from Dropbox it's not such a big deal I'm going to get this whole thing on the screen this is the this is all of the algorithm code that we have that's it will actually leave us with this file one more time to add in the bulk a piece later any questions most of you start if you really think about the process you start in a notebook you sketch something out then somebody some Engineer comes along and puts it into a Python script what's the next thing well putting to a Python script doesn't really help that much because at some point you need to expose this to a web and we know what one argument that we had over lunch was whether you're really buying into something when you choose rest and JSON as your RPC mechanism and I made the incorrect assumption that one of the things that you have to buy into in this talk one of the things you have to agree is a right way to do things it to say that you have to buy into rest and JSON API stew is to say that you have to buy into breathing air and drinking water that it's such a fundamental and ubiquitous part of our lives that it really doesn't involve that much buy-in I would revise that and say maybe the next part of this the approach that we're gonna have is more like buying into eating cereal for breakfast you know you go to the store you go to Tesco's you go to the aisle for breakfast food and there's cereal there it's just a choice a lot of people do there are many better ways to structure this there are many different RPC mechanisms you may have your own opinion of them but JSON rest that's one of the most common things that people ask you to do so let's do that next part and in order to that next part we need to use one of our first actual non data science library's flask it turns out if you're using anaconda flask is already installed you don't have to install anything at this point if you are not using anaconda you're gonna have to figure out how to install flask ideally pip install flask and you're done good luck for me I'm using anaconda here so it's already installed so I'm going to create another file and this other file is going to be called REST API py an arrest API dot py I may just write a little flat cap that will serve an API will serve these results over some-some HTTP API so again I'll make this a Python 3 script and one reason that we can use flash for this so one nice thing about flask is one thing I didn't want to do in this tutorial or I wanted to minimize in this tutorial is what I see as an unfortunate case in a lot of tutorials even in the flash tutorial especially in tutorials of tools like Django which is the first three sections are well why don't you install this and set up your file structure like this and create all these empty files in all these directories and frankly you don't care about that what we're going to do in this in this tutorial we're going to write everything in as few files as possible we're going to take what in a normal project should be three files I'm going to write it in one file just because we want to get something working quickly and honestly done is better than good Slask is actually a pretty good tool in that what you'll see is our use of flask going to be very very minimal there's not a whole lot that we have to know I'm not a whole lot we have to buy into to make this work so we'll import flask from flask and we'll create a flask app we'll create a name block and in this in this main block this name equals main block we'll run our app with debug set to true our app can't really do much it can't serve anything to anybody now if I try and run this app I'll do em Python 3 REST API you'll see up port sardian use maybe I'm running the answer in the background ok because it will default to port whatever so just make sure you're not running anything else at the same time let's try that one more time there we go but I flask app can't do anything can't serve anything to anybody but it's sitting here on my machine at localhost 5000 if I want to access it can't do anything very interesting yet now we want to actually do something interesting with this and so what we want to do is we want to be able to I don't know like actually run the algorithm so we'll pull that from our will pull up rocks from algorithm and we'll just create a simple input app that route and we'll set it at the at the root and we just have a function and it'll be called for now we'll call it put task because we'll use that name just to minimize and we'll just go into the we need one more import from flat we need two more imports from class requests and JSON fi requests gives us access to all the things that were sent as part of the request namely the JSON body of all the arguments you send and JSON fi allows us to automatically return some results in JSON so we'll just say a equals request a sign of a B because the same thing for B C equals the same thing for C we're just going to unpack all the arguments that we passed in fact maybe F is the first one that we want and thankfully we have access to a fantastic technology called cut and paste and then once we have that we'll just say our result is equal to JSON of Phi actually we'll set our response is equal to some result and we'll call approximate with a sa BCD and I will actually do well get the size as well we'll say request dot JSON that get of size and if it's not provided will default to one hundred stereo and then we'll just return JSON of Phi of the response that's it this is basically the core of our flask app now we need to add a couple more pieces to this but before we do that I want to show you this in action now to show you this in action you'll need to actually trigger this API now you trigger it through your browser is going to be a pain so here's what I suggest we're using whatever tool you have ideally anaconda do a pip install HTTP this is a tool that is way easier to use for testing json api s then curl it'll allow you to do something like the following HTTP GET HTTP localhost 5,000 and let me pass the parameters so my parameters would be f is equal to what was its square root of 4 minus XS squared my a and I'll put a colon here to make sure that it's a interpret this is an HT 2 pi ism to make sure this is a sent as a number or not a string just remember JSON does have some rudimentary types and I'll leave this one on the screen after I run it huh where is this running we don't have our flash server running which is why it doesn't work here is where your life is going to become miserable you need to have a win a window or a tab or something else open and in that tab we're going to open five panes letting all of the five things we need to actually make this stupid task work and the first one is running our back-end server so here I'll do again M Python 3 REST API our flatgap is up now if we go back to where we're writing the code and we try and query our flask app you can see it gave us our result 0 so let's just make sure we have F ABC oh there we go that was a typo and you can see we're now querying this over a REST API that's basically the core of it now what I'm gonna do is I'm gonna give you a minute or two just to make sure that you're caught up or to copy from Dropbox make sure you have flask according the background we honestly need to have five separate services running to make this example work it's just a nature of how this works when you were when you actually deploy this to production there are a lot of tools that you'll have to simplify that from docker to systemd but when we're kind of hacking on something locally it gets ugly so hopefully you have some window management solution like T MUX or screen to make this easy who needs another minute to get this ready HTTP HTTP ie or you can use curl just that curl is ugly and I'll just so that I can do it I'll show you this is what my call using when you install it the binary is called or the executable is called HTTP without the IE and actually if you look in the drop box I'm going to I'm going to make sure my install notes if you look in Dropbox I'll put there's a if you look at Dropbox in the backend folder there's a list of all the things will so this is the one that we installed we'll install these five things before the tutorial is over there we go now the problem here is this folks one the built in HTTP server the mechanism by which we're actually serving requests HTTP requests via flask the one that's built in you don't see anything beyond just this app dot run it's single threaded and I don't know if they can service multiple requests simultaneously when you actually deploy it to production you'll use some tool like G unicorn or gosh knows what G of N or or there are many many choices and you'll go and you'll research them to actually make this work for large numbers of simultaneous users one of the problems even if you do that with large numbers of simultaneous users is when we bump up that size too high we're going to block so whatever worker is servicing the HTTP request is going to just sit there and block and give a finite number of workers and whatever is making that request is going to block and not be able to get anything back so we need to make this so that we have a job server so that when you submit something to our system to our platform instead of just running it it creates a job that runs in the background run separately and then we query is that job ready or not and if that jobs ready then we pull the results out so let's see how we do that in order to do that we need to install two more things actually we need install yeah two more things we need to install celery we can write our own job server if next year I'm invited to speak again maybe I can show you how to write your own job server using tools like using curio or async i/o it's not that hard to write your own job server the thing is you won't be able to get the same quality of tooling that existing libraries will give you we don't have time to write our own job server so instead we'll use something off the shelf we'll use celery is this task manager slash job server and we'll try and write as few lines of celery as possible so we won't use any of the sophisticated features and celery knowing that potentially certain features of our job server that we need that are very sophisticated may not even fit into celery and we might have to write it from scratch eventually so we're just going to do the simplest thing it takes to get this working at this point all of you should have a working flash gap if you don't have a working flash gap I would suggest that you save your work off to the side and take a copy of what I have in the back end folder to turn this into a celery app is not that hard but we have to actually install celery so in my side window here I'm actually going to install celery and I can Conda and I can pip install celery itself but one thing about celery is and I'll explain two very very based in very basic terms order this celery is a separate process to which you can submit tasks it will run tasks it will be able to tell you when a task is completed and will be able to send you the results of that task there are two important components for running celery the mechanism by which you submit jobs and the mechanism by which you get the results back the easiest mechanism for submitting jobs is a simple message queue message broker called Redis we will have to install Redis and run it as a separate process thankfully throughout this entire tutorial we will never have to restart Redis we may have to restart our flash server if we change the routes too much but we'll never have to reinstall Redis so you can just get it starting somewhere in the background if you're on a platformer you can app get Redis app get install reddits or something like that then don't worry about this if you're not and you're going to install it the way I'm going to install it which is via anaconda then you're going to have to manually start it up and you'll know if you haven't started correctly so the first thing that we'll do is we'll pick install celery using Redis the second thing that we'll do is we'll Conda install reddit now in order to countenance tall Redis it's not part of the basic Condor libraries so we'll need to use a line that looks like this now if you don't know where that line is if you look in the backend folder and install that Sh here all the installs we're gonna do so we're gonna do the one that says Redis next on line five and hopefully it's in the background with all of us jamming on the Wi-Fi at the same time we might have to switch to my existing environment it's fetching the package metadata solving package that's I'm like the person who sits in the passenger seat of your car and reads the signs as you're driving it must be incredibly irritating it's solving package specifications next must be a lot of Patrick specifications that has to solve let's go ahead and install this should take about another 30 seconds and then we have reddit result so we're waiting for this Thank You stakeholders come online come on yes I can't actually let me do that here I'll do it in the here what I'll do is I'll do this I'll do his file equals history shell and so let me let me run the so let me do so you'll have to you'll have to read through them because you don't have this M program that I have but what was it M HTTP GET HTTP localhost 5,000 and will to get requests even though we don't want this to be a get request eventually square root of 4 minus XS squared and a was equal to 0 B was equal to 2 C was equal to 0 D was equal to 2 right and that should give us a result let's see yep that gave us a result if you look in Dropbox there should now be a file called history yep called history and it should contain I let me make that history on SH because that way you'll that way I don't have to download it this we got a seiche ok let's see that go back thank you that's a really good I was really good idea so we see refresh this history SH and you should see that's my my Z shelf history so I'll just keep track of all the history in that one terminal so you'll at least be able to see them even if you can't cut and paste please just cut and paste from where you see the word em to the end assuming your environment is set up correctly okay we've got Redis we've got celery installed let's start up Redis so I'll create another tab here and starting Redis is pretty easy you just do Redis server and it's that's basically it you will never touch this again open a terminal start in the background Redis won't crash it doesn't ever need to be restarted we just need it running if you ask get installed it you'll be fine because it'll be running as part of whatever your nips mechanism is your system D or Y or heaven forbid X I net D or whatever now we're going to write another Python script and that Python script is going to be called worker dot py in worker dot py will actually use celery in order to implement these tasks this is a very simple script it'll only be five lines long celery has a very easy learning curve to do simple things from celery will import celery from algorithm will import aprox will create a celery app and we don't really care what the name is but it's called worker and we have to pass some parameters the backend is how we get results coming back at us so we'll just have the results via some RPC mechanism and the broker is the queuing mechanism by which we send results in so here's one small thing that you need to be aware of when my Redis server started at the very bottom here it said the server is now ready to accept connections on port 63 79 if you install this via brew or apt you should be running on the same port but there's always a chance you're not so there's always a chance that you're not so just check that line hopefully you are and we'll just have to broker to that then we just need to create the celery tasks and it's very easy we can just say integrates equals into great equals apt task of approximate or but we don't really need to do anything in the decoration typically what you'll see is you'll be written it like this app task def integrate blah blah blah but we can write it a little bit shorter because we don't need to pass any different arguments so there we go we have a celery task that can be launched from a worker that's it actually let's make this a little bit better because we're going to have bugs in our code later so we'll actually do it the other way so we'll say app task def integrate some args some keyword arguments we don't care what they are we'll do a try aprox the args and the quarks and if there's any exception we'll return a return nothing that'll that'll save that'll save us from some things that potentially will crash later so you could do it that way or you can do it the easier way the easier way would just be integrate equals apt if you don't care if it crashes or not gain do it that way Oh okay that's it that's all we need to know about celery trust me it gets worse from here you a lot of you are already thinking oh my goodness it gets worse and that's it and unfortunately that's the nature of the technologies there's really not a whole lot we can do we are cutting as many corners we are cutting an enormous number of corners but it is very it is not unreasonable i I don't think it's unimaginable that somebody in your work environment would come to you and say okay we expect this to be deployed in some web-based form we don't want to hire somebody else to do it why don't you figure out how to do it and you're required to do all of this stuff and that's the purpose of this tutorial to at least give you that ability to show you that it's possible to hack your way through it without completely giving up because when we get to the next section of the tutorial it's going to look pretty hopeless there's any JavaScript ecosystem is a lot to learn about ok I should I should be somebody say something really positive and optimistic I'm going to go into my back-end and I'm just going to I'm going to start I'm going to start my celery so rabbit it's very simple to start the celery worker it's just there's a there's a program called celery and we just give it the name of the Python script or the name of the worker which is called worker then we tell it it's starting a worker so it says work or twice there and we'll just set up it's called worker it's this one right here yeah so it's the app it's the app that it's the a here is the app that you're using and the app happens to also be the module name maybe I should have named it something different to avoid that and I'll just say the log level is a debug level there we go and you can see it can't quite connect to my Redis server so let me just make sure I didn't do anything wrong here 63 79 63 79 let's try this one more time oh I misspelled localhost by the way I hope you'll all join me for the pub quiz tomorrow where you will be required to spot small bugs like that in order to and impress your colleagues with how well you know Python here we go so we have now three components running in the top left we have our flask API in the top right we have our Redis queue and the bottom we have our celery worker now once you have that does anyone need a second to get up to speed with the celery piece to start the cell yerker I'll put it in this one I'll just put it as comment here but it is a celery actually put it in the file itself run with celery - a worker worker log level equals debug so get that and I'll see how to starts Redis as well so Redis is just ready server so with those two pieces I'll give you two minutes do you have from sis import RV at the top of your right here double check and if not set yours off to the side and cutting plates for me because we're a little bit behind schedule already we can run a little bit over we can run a little bit over because the next part isn't so you think this is bad this is actually this is the part that was supposed to be like wow it's so easy because because honestly the website of this the tool we'll use here we act is a whole heck of a lot friendlier in the way we used to do this with Django and ginger and templates but it's still there's a reason why people do this as a full-time job maybe at minimum maybe at minimum what you'll get out of this tutorial is some respect for those JavaScript monkeys that we all make fun of they live hard lives too we're going to improve our flask app so we're going to do a little bit of code turn here we're going to improve this and the way we're going to prove this is instead of running this task directly we're going to start a celery task that we'll run it in the background and we're going to add a way to query whether we have tasks start or not so what we're going to do is at the top of our script right after this we'll just create an empty dictionary and now represent all of running tasks what this means is every time we restart our flask server it has no permanent storage so all of our tests and all of our results will die which is good because we don't really need permanent storage for this example and because it allow us to clean this plate by just restarting our flatgap so this is a dictionary that will store all the tasks and I should call it tasks with an S well so all of the tasks that we start and sell you all the jobs as it were this here is called put tasks really it should respond not to a HTTP GET method let's do an HTTP put method so I'll say methods equals what to make sure that if you need to actually sense up the end you put it and what that will change is when I'm running my HTTP command here instead of doing a get request I'll use the put verb otherwise nothing else has changed you can see it's the exact same thing I just use the footwork the reason I do that is I want to reserve the get verb for something that makes sense to get ie the list of all the running tasks so I'll say app dot route and I'll route on the index and I'll say methods equals get and I'll get I'll get all the tests I'll call this a function I'll give this I will call this function list tasks and this function will list all of the tasks that I've been submitted I'll say tasks is equal to task ID and then whether or not it's ready tasks start ready for task ID tasks in tasks items little little simple thing there and I'll just chase on apply that and return it there we go that's it and here the only other change I'll make is right here instead of calling this function directly I want to I want to do a delayed evaluation of it okay and that's as easy as just saying a proc stop delay instead of call the decorator that we used actually it's called integrate delay because we change the name of it integrate that way and instead of getting a procs from the algorithm module we'll get from worker import integrate so there those are the changes but what you can see fundamentally is instead of having one function that runs eagerly we have a function that gets scheduled and the dot delay is your way of scheduling is in your celery worker that's it the major changes are here here and this piece here yes that's absolutely correct we need to soar the task thank you let me do it this way then task ID equals thank you that was a really good observation test of task ID equals that and the result will just be the task ID pack back to us there we go thank you so I'll maximize that I'll give you a minute or two to either cutting paints from Dropbox who's given up completely and there's now cutting and pasting every step of the way it's okay okay it gets worse from here oh it gets me oh my goodness javascript all right that's fine that's there yeah that won't be cool move so if you're going to cut and paste from Dropbox just wait till I run the code to make sure it works it might be a good hint I'm a very imperfect person and make many mistakes to err is human to forgive something I can't look at it if you're returning something from reddit you'll get a pickle here for Mattituck returning your return it's a dictionary right [Music] maybe we should start with this how to install virtualbox there you go there you go for those of you who are managing to follow along this far even if you're struggling I'm extraordinarily proud of all of you I don't know if that means anything it shouldn't but I am as I understand how difficult and how much I'm asking of you here okay what I'll do just to keep this on the screen is I'll split my window the other way to do to know that you could split your screen so many ways and let me do one thing let me do my little sis tiling poles kind of that there we go and I'm going to show you a CTP get localhost 5000 and this will give me all the contacts none I have no tasks of course let me put a task I'll put a task and I'll say my F equals give me an F maybe we'll just do our square root for minus XS squared a is 0 B is to C is 0 D is 2 we'll put a task it told us our result ID was 0 we'll see it's ready it's running in the background and if you check your little log here you'll see hey celery got a task and I did something how about that now the important thing is we won't actually get the stupid result back right we want to get some results out of here so we need one last end point the last end point is the end point where we actually get the result back and it takes one argument which is an integer it'll be called task ID and it responds to the get method the HTTP verb the rest verb and it's just called get task takes a task ID and all it'll do is it'll say response is equal to and we'll just encode the task ID in there because it might be useful oftentimes I find when you're writing these kind of AP is returning is up much information about the requests in the response is very helpful because sometimes in the front-end app there's a loss of correlation between the requests that you made in the response that you got so trying to tell people not just the answers their question but the question that they ask ends up saving you from kind of coding yourself into a corner in some places does anyone have a friend who when you're texting them always responds like three texts too late and you can never tell what they think like you'll ask them three questions and they won't respond to them in order oh if only human beings could be as wonderful as computers once a task equal task ID will pull it out of our little tasks data structure and we'll do this in lieu of something like an actual database or some persistent storage and we'll say if the task is ready we'll say the response we'll just put the results in there and to get that response is the second part of selling a third part of celery need the first was our five line celery app the second was we get this function with a dot delay callable on it that allows you to schedule it and the last part is the dot get call which allows you to get the result now the dot get call can take a timeout except since we know the task is already ready we don't need to time out of it and then guess what we'll return JSON of Phi of our response now thankfully I'm really surprised our flask app has never needed to be restarted some of you may have to actually restart your flask app now cross our fingers if we do a slash zero uh-oh something broke so let's see did it did it did it reboot no there's a key error oh yeah sure you're right you're absolutely right every time that's a very good point what was your name Isabelle made a fantastic point one great thing about running flasks with this debug equals true is that those hot loading of code every time you change this to the best of its degree it will reload everything so notice we never restart that app except it will reload the module which means it will reload our of storage so the reason that failed was no such task existed we might make this a little bit safer so that if you ask for tax it doesn't exist you don't get something back we could do that pretty easily I will just be making this uh I don't really want to do that so the I like my code to be as unsafe as possible so we'll put our tasks here will list all of our tasks it's ready and I'll see if we can get it how about that we have now completed a flask API with a job server that wraps our algorithm and then the next part of this tutorial is how do we actually make a front-end website that uses this when you are actually doing this in production hopefully you'll have more than 90 minutes to do it trust me it's doable in 90 minutes although to actually prep all these pieces together took more than 90 minutes to do it may not look like I spent any time putting this together but but trust me it took a little bit of time to get these pieces together at this point if you haven't been able to follow along and you've given up I know who you are I can see you having put your laptop's down go to that back-end folder download the back-end folder as a zip unzip it and just start up those three tasks the flask server Redis and your celery server and you can continue on with the next part of this assignment because you'll have exactly what I have there'll be no problem yeah we'll do the I'll do the bouquet stuff as supplementary because we're running low on time so do we have a talk after this no so what we'll do is we'll do just a simple thing and then there was end in the last part it's only about 15 minutes if you want to stay a little bit later I'll show you how to add in the bouquet stuff where you actually have the response pop up that beautiful little here's the dots okay so we'll cut we'll cut that corner for time so those of you who've given up don't give up don't give up yet give up in like three minutes once you learn about JavaScript okay okay flask app I'm going to put that in the outer in the rest api I'll show you how to do it it's very simple this one you just start with just Python 3 and then the name of the script rest api WI that's it and I'll put these three let me let me put this let me put these three lines actually if you look at history SH history is H you'll see that's my history of all my things so it's in Dropbox as well so it's in the top level everything on this little side panel will be in the top level deck y'all having fun okay here's the thing I got in trouble earlier today by saying something that I will repeat that is maybe unnecessarily provocative but I have given a lot of talks at PI data events this is my 24th PI data appearance No maybe 20 seconds I'll count I'll tell you tomorrow and I mean this as gently and as lovingly as possible but data scientists do not care about code they just don't they care about their analysis they care about their research they care about the immediate business tasks of trying to solve and code is just one of the things that they need to do to make it happen which is why the moment you meet a data scientist who has somehow convinced somebody to pay the money to just work on models to just write into all do all their work on pen and paper and maybe only use a computer to code things in latex so they can publish it for people they're happy as can be because you don't care about codes now the PI data audience is a little bit different you're a little bit more caring about engineering but I have to say I have read the code written not by anyone in this room I'm sure you're all brilliant capable technologists but by many of your colleagues and I'll say if you care about code well I don't know I've read your code I can't tell that's ok because the economics of the way in which we work is not really set up for you to care that much about that code there's only so many things you can care about in the world and what they're going to care about is your main skill everything else is going to be secondary to that that's just the way it is there's nothing wrong with that when we look at javascript in about 30 seconds from now it's going to be overwhelming because in order to actually be effective in JavaScript there is a lot that you have to care about it is just it's terrible there's a reason why why JavaScript programmers make as much money as data scientists because there are a lot of things they have to worry about what we're going to try and do to the best of our ability is eliminate as much of that as we can but there are certain things we simply cannot eliminate I bet a bunch of you have written JavaScript you know back when I first wrote JavaScript and what did you do to write JavaScript well you're on your Windows 95 computer and you opened up notepad exe and you wrote alert hi hello world and you order index.html and you put that little code in there and you refresh the browser little alert window popped up and you're very proud of yourself maybe if you're a little old a little younger than I am you start with something like jQuery this is simply not the way in which modern JavaScript single page apps are written these days and I'm not an expert so if somebody wants to give a lightning talk tomorrow and say why I'm wrong I would be more than happy to be shown to be wrong because this is only the impression that I've gotten as a data engineer dipping his toe into this area as a matter of having to actually be paid to do work in this area the way that javascript is developed these days is akin to how we develop in C++ or Java you write some code then you have a tool chain that converts the source code that you write to what actually runs in the case of what we're going to look at here the tool chain uses a very popular tool called Babel we are not going to write JavaScript that will run in your browser in fact all of the code that we write will not run on your browser it will have to be transpiled from a JavaScript scanner done on every browser supports we're going to be using the 2016 standard ECMA six I believe is the formal name plus some stage 0 requirements that I don't think we'll use the will include anyway so plus some features that not everybody even thinks will get into the standard at all plus a technology specific to reiax called JSX that will allow us to expose one of the nice things about the react technology which is this component driven design we need a tool chain and so when you're writing C++ and Java in school you learned oh I just typed GCC and I produce it in executable and then waiting to work they said well you can't actually do that you have to have a make file and then somebody said well you know what make doesn't really work that well for Java let's have an aunt in maven the exact same thing exists in the JavaScript community except the tools are not as old as the tools we have in the C++ community or the Java community and it's not clear that they will even exist for the next five or ten years that said even though none of the stuff will exist for next five or ten years doesn't make a difference as to what you need to do now when somebody asks you today to write the code word that's right I'm going to try and minimize the number of things we have to learn in the remaining 20 minutes or so that we have in order to do that you'll all have to cut and paste at this point there is no choice because there's simply too much so I'm going to show you what you're going to cut and paste you don't actually have to cut and paste per se there is a folder in the Dropbox called front end in this folder I have the most minimal set of things that we need to actually make our single page app when you go into github and if you search for react template you'll see a million or one react templates everybody has one the problem with those react templates or just like all the Python Django and flash templates I see which is they give you a template that does 50 things of which 49 you don't care about seriously the only things we care about right now are building our code to run the browser and having it automatically build every time we change the file so that we don't have to do something in a side window we don't have to trigger the compile directly everything else we don't care about continuous integration don't care about linting you don't care about testing we don't care about and the unfortunate thing about a lot of javascript tutorials and templates and how-to is that you see is you're going to force you to buy into everything and that ecosystem has not coalesced around any one specific combination of tools yet so each one will require that you buy into a different set of tools the tools we're going to use here are not the best tools we'll be using NPM will be using grunt will be using Babel there may be alternative tools in each case what I'm going to do is I'm going to use each of these badly enough and quickly enough that when it comes to the point where you say grunt sucks or rather gulp sucks I'd rather use something else you can swap it out and you haven't lost that much time I'll show you what finals we have here static is our actual what we're going to serve the actual static this is the HTML you used to write many years ago except it's been written for you we will never touch this file this file contains a basically empty HTML document and our react code will actually load the app Internet we do not ever have to modify this file this file there's a CSS a style that CSS that contains a single directive just to make sure that our code pops underneath the page header will never change this for this entire assignment the the index.html also includes a couple of JavaScript requirements and CSS requirements to make the bulk a part work and to make basic styling with a CSS framework called bootstrap work even at something as simple as CSS frameworks there are new choices appearing every single day I choose bootstrap because I can show you an example quickly with bootstrap and you don't have to know that bootstrap exists but it's maybe not the best tool for all uses but we'll get a lot further making these bad choices than if we get paralyzed by having no choices source is where we're going to put the JavaScript source I have tried to extract some of the components that we need to write to a file called components jf' because they're just not very interesting for the purpose of this example the rest of these files I'll explain to you package JSON package JSON is your JavaScript NPM equivalent of requirements txt we install we'd like to to kondeh install in a pip install if we need to do the same thing to make this work we'd have to install all these libraries javascript does not come with a rich standard library and there is no as far as I can tell there's no equivalent distribution of JavaScript or distribution of NPM like the res anaconda anaconda gave us flask without us having to install it it's not really the equivalent here so these are all the dependencies we need just grab that file this gulp file this is not a very good gulp file but what the gulp file will do is it will build our code for us every time we change the file so we don't have to trigger the compile step manually it'll save us a little bit of tapping back and forth and then finally this babel RC now usually I we don't have to look at what's inside it but it gives us the rules for how we take the code that we write which is not in browser runnable JavaScript it'll be in a language called es6 plus j sx and actually manages transpile so download this directory unzip it somewhere and we're going to go once you have that unzipped the very first thing you need to do will go into my front end directory yeah is do a we need to install NPM but I should not have to close that tab that's so we need to install all our node modules but in order to install node to actually get the dependencies to run this code we need to install we need to actually in sort of to run NPM we need to install node and so if you look at backends install that Sh maybe this isn't the best place to put it we're going to install this dependency next we should install two dependencies will do this no js' dependency I'm going to do this flask ors dependency and we're going to do it we'll have to talk about actually we'll just do this one first I'll talk about what that class courses it's a really stupid little you know why do we have to do this so we'll install node.js and so the conduct fords includes the ability to install node.js this is an old version of node the current version of notice seven ten this is six ten so I think it's at least a year year-and-a-half old when this actually makes a difference to you I expect you to go out and figure out how to install node by some other means for the purposes of this exercise worse is better done is better than not so no it will install it into your contents of it so that's the other nice thing about this tutorial everything is in your con environment so install this if you do not install NPM you cannot continue with this part if you don't have MPM installed okay so this isn't this is install this it doesn't take that long to install and then we're going to see the great thing so javascript has no you know when i do corporate training for people who write code in java or c sharp and I try and teach them Python I tell them you know Python comes batteries included there's a really rich standard library and these days the Java standard library is not bad in c-sharp stand below is not bad javascript standard library is very very soon it is exceptionally thin and so when we do our next step which is NPM installed we're going to install all the parts of that we'll make up our own personal java standard library and there's a lot of stuff that's been installed not just the stuff in that package about JSON but an enormous number of dependencies for all of these so do an NPM install and that will give you a running node environment they should be ready they should be ready oh so your tasks are just so there might be a bug in your algorithm code where it's crashed or it's especially for running on Windows I don't know if numpy dot random that uniform I mean it could be just taking a long time to generate the it should be they should be ready okay so here's all look at this this is how much stuff we actually I'll show you this is disk use human summary of node modules so we had to install 80 megabytes of standard library code all JavaScript to even do the simple hello world we're about to do don't let me discourage you about react it's a really good technology you think you think you're having a hard time the next part is going to be introduced we need to create a react app and I'm going to build this up for you but we're going to go very quickly what we're going to do is first in the front end directory in the front end directory in the SRC directory we're going to create a file called index J s okay I'm going to add the first line you're going to see that if you have any familiarity with any language that's Algol derived like C C++ even Python kind of counts react syntax more or less you can kind of read it whether you can write react whether you can read you can read a JavaScript syntax whether you can write JavaScript syntax or ESX syntax can be a little bit harder because in some ways JavaScript modern JavaScript is better than Python it has some syntactic structures that are really nice that I really wish we hadn't tights on but in other ways it is significantly less regular than Python in Python if syntax works in one scenario it'll work in all equivalent scenarios in JavaScript you know you can write that you can write this and when you have syntax looks kind of similar to this you can do different sets of things don't ask me well I can tell you why but don't ask me why next we're going to import render from react sum and then we're going to add just a simple part here will render now will render an h1 that says hello world and we'll do a document dot get element by ID now what you might want to do is you might want to have my code open on the side while we go because this parts going to move pretty quickly now with that what you'll see is in my front end directory I now have in my source directory I now have oh yes it will it'll try and synchronize all 80 Meg so let's hope Bloomberg has fast Wi-Fi let's check to see how slow that's going let me see if I can turn that off that done there we go so let's do general bandwidth toxie's account selective sync tutorial front end up now we want the front end but we want source and static but not node much less thank you and now we don't I won't do that because I don't want to accidentally stuff direction so give it a second it'll ah yes it will in fact sync all admx 80 max isn't that much anymore Oh trust me yeah you remember the days when a when a HTML apps were in a you know 30 kilobytes of HTML was a lot mm-hmm even the react apps are I mean prepare for just be happy that we have caching so I'm saying okay give this a second to somebody somebody watch it and let me know when I when the when the index dot JSP and I'm going to do one thing off to the side and there's this is the second-to-last service you have to start in the front end directory and this one's a little bit tricky to start so in the to do this history goals yes that's correct Oh see that wasn't so bad thank you Bloomberg okay in the front end directory in the node modules directory there's a dot bin we're going to have to use a file in that gulp and we're going to just say gulp watch what gulp does is it's going to be building our source code seance you said if I don't know why it comes up so big every time I build the code it will pop that up and I'll tell me it built so my react app just built and I can serve it which means I need to serve it now I already have a flask server serving my API I need a front-end server to serve my front-end code I have included this for you it's very badly written and it's called HTTP server dot py just run it as a Python script just do python HTTP server it will run an HTTP server on port 8000 that serves just the static components when you actually deploy this you'll probably use something like nginx and you'll have nginx dispatch for the API endpoints to something like G unicorn and then directly serve the static content ie the HTML the JavaScript and the CSS so once we do that now I can go to localhost 8080 HTTP server and just put static so to this thing let's you tell what directory to serve so we'll serve the static directory so we'll do this one more time hello world that's our react app so we can see this is being generated by react now that we have this we can start iterating very quickly react works wrong what you might call a component model so what we'll do is we'll build components representing the different parts of our program I have a component that I've created for you called layout so import that and a component called navbar from components and I'll replace this rendering with layout where our main is equal to a component called main what you can see here is what's called JSX JSX is the ability to transpile in a javascript file something that looks like xml and so what we have is these JSX tags surrounding things which are components this is a layout component that we can write this layout component takes ask things which looks like active you it's called main but in the react terminology not attribute there are properties and it has another property called navbar which takes a navbar element that I was going to create and it renders now what's going to have what's going to be a problem is I haven't defined main so I'll show you how to define main I can define main it's just a function that returns a HTML tag so there we go it built it'll say I can't load and we'll see what the error is I've misspelled something I know it should be from dot components because it's in the I it's in the same directory right that's a good that's a good question because I see why it's should be let's see which of these let's see which of these is a failing so this is give me one second this part is it is it it might looking at this no wrong place now should be in thought components so we'll do this which just what you do the layout by itself this is yeah that's really interesting what did I do wrong here I hope I hope the components I gave you I didn't give you the old version of this poop I give you the old version okay yeah there's there's one line missing so just grab that file again that lines missing okay now with that built oh and this also this also has alignment secure - so you can have to grab that again because this one had some fixes so just grab component don't try and fix that just grab just grab components let us again in fact you might want to just start I haven't done anything that interesting so far okay so here's our layout and then let's - main equals main now far equals now form and this is basically the simplest layout that I could come up with there we go we got a hello or not and now it looks kind of like a real app now we're going to turn through a little bit of stuff our main needs to actually have a form that lets you put in those fields so I'm going to do this very quickly and it wouldn't be such a bad idea if you cut and paste from where I go this is a function that returns an element you can write react components as functions or as full classes I'm going to write this as a class because I need some additional functionality JavaScript now has object or a proper object orientation it still uses a prototype model but you can do things like extend components react is based around what you call a life cycle so it has specially named methods like renderer component did mount component will unmount to indicate where this component is in the lifecycle essentially react tries to make your life easier by keeping everything stateless state flows from the top level component down components have properties and some internal state that they manage when they are created when they're mounted onto the documents onto the Dom you have the ability to to hook into that when they're rendered you have the ability to hook up to that and and update what they look like when they're rendered so when this thing renders its going to do something very simple I'm going to cut and paste a little bit of this it will it will return just that and here I'm going to do a little bit of turning hello world like that so now what's that you'll see nothing interesting happened now what I'll do is I want to have a react class that represents my pending job and I want to have one that represents my completed job okay and I want to keep track of what my pending and completed jobs are for this instance so just like a Python function has an it a react component or Python classes internet a react or a JavaScript class has a constructor I'll call the super constructor and I'll set the state so this react component has some internal state now one thing that was originally to be in this tutorial when it was a three-hour tutorial was a technology called Redux and took note and the technology Redux and mob X are things which you'll almost certainly want to reach for the moment this app becomes more than ten components large before when it comes like five hundred five hundred lines of large these are mechanisms for managing your state or managing global state better between components you will not see the motivation for it until the app gets large enough and it won't get our Jennifer this example in the render step I now render all the information I have for this component so what I'll do is you can see this is it kind of nice this unpacking syntax we have a Java Script I'll pull out the results in the pending jobs and I'll just do a little bit of magic I'll call this my oops changing tag integrator and I'll just and I'll have some form here so I'll just put a little placeholder for the form for them and then I'll show you all the pending jobs so I'll map pending this one this one is a little hard to build up because we're running out of time do you mind staying other 5-10 minutes okay so here I'll map all the pending jobs this is a JavaScript dictionary and so JavaScript exteriors you can actually map over the values and the keys unlike Python dictionaries which map only over the values so here you get the value and then the key and I'll just create a actually I won't even care what I take one thing is JavaScript functions don't enforce their arity Python functions if you pass too many vary too many arguments it'll fail JavaScript functions can actually be passed more arguments than they take and the rest just get thrown away so here I will just create a pending element for each of my pending items and I'll create a result element free to my result items ok now this map function you might wonder where does it come from javascript is a semi functional language one thing that's missing in javascript is that some of this stuff is kind of hidden on the built-in objects and so there is a tool in javascript a library that would really encourage you to use that will make your life much easier it's called lodash it gives you the ability to do many of the operations you take for granted in the Python standard library map filters sorted enumerate it gives you that back what you get used to when you're writing Python code using the built in you don't have in JavaScript but lodash gives it back to you there's another similar one called underscore that gives you a lot of these helper functions now I notice each time that popped up red react components must always return a single component or element so every time I'm trying to return multiple things I have to wrap it in either div or another element now that worked there you go there's my integrator in my form let me make this a little bit richer very good thank you so here this will now take a when you write a react component as a function it takes its first argument the properties that are passed in I'll pass in a property called ID and I'll make this component very simple it'll just be an h2 that has pending and then the ID and you won't see that because I don't have any pending jobs yet now in order to actually create pending jobs I need to create a form element this is something that react where the react model kind of breaks down form elements because form elements have their own internal state like what's in the form element and react manages state top-down from parent containers to child containers things kind of break oftentimes what you have is that a form as a child container on a page needs to effect something over here and you either have to reach for technology called like mob X or Redux or ideally you find a nice library for doing forms now the library that we're going to use for doing forms was picked something out of a hat one thing when you start playing with react that you're going to realize if there is a normal ecosystem of react components that you can access and oftentimes a job that would take you an afternoon to do or a full day to do you just grab a component use it and be done in 15 minutes that'll happen to you all the time there are kid hub is littered with react components some of them are very good some of them aren't very good you're going to use a react component that you're not going to immediately realize is bad and you're in your birth life it you cannot avoid this it will happen even very well even once with really nice github readme files and you know unit test integration some of these react components are just not written very well and it can be difficult to assess without using them whether they will actually fit into your structure especially because you will not have the upfront knowledge of the react way of thinking your kind of trying to hack something together and it's going to happen so just be prepared that for all the time that you save with what we've done it we've done an enormous amount of work in a short period of time you're going to lose some time we ran some code turn by picking bad components so let's pick our first bad component and the bad component we're going to pick and I'm going to grab this one I'll speed up a little bit just so that we don't take too much of your time is called form Z react components and what it will do is it allow me to create to very quickly create HTML forms where I can grab the information from them so I can remove this placeholder here and I can replace it with a form now I'll show you what that looks like because I'm going to cut and paste this from my own notes there essentially this looks very similar to the HTML that you might be familiar with for creating forms actually let me wrap this in a component request we'll find it like that and then let me just wrap this in a component and I'll show you what that component is but we all look into it in great depth so these three components really represent the majority of our app the form itself and you can see it's one of these forms that takes some function when you submit it a bunch of inputs F ABCD and the size and a button that you press to submit so now that we do this we need a we need a on submit button here and we'll just have it two nothing interesting now when we do that finally we have their stature that looks like that was about 10 lines of code sadly I wish we could have written it line by line together but I don't know that we have the time when we submit a job depending jobs will show up beneath and the completed jobs will show up the needs as well notice that this looks like a nice form you can see this component we kind of picked out of a hat but for this many lines of code we ended up getting something that looked pretty nice this is the story with react there are a lot of components that will save you a lot of time they might bite you in the but later but overall you're going to be able to produce things very quickly that will look very professional and will work well which is why there's some motivation for using this you'll be able to do a much better job than you could with the old style of Django and ginger templates my a my in my opinion is the last thing we have to do is we have to wire up this form to actually submit the HTTP request in Python we use a library called requests in JavaScript there is a similar library called super agent and we will import requests from super agent will also import numeral from numeral Java scripts standard library is very weak it doesn't have good tools for parsing things like the numeric input or parsing things like dates this is why you may see popular libraries like moment jf and numeral j s for doing these fundamental tasks will use numeral so that when you give some input I'll just handle that parsing of that input to a float automatically without having to figure out is it's an answers as a float you know do I handle do I handle scientific notation all that unfortunately that's not as easy to do in JavaScript as you'd like who likes JavaScript so far it's actually pretty good trust me so here this on submit is a function that takes our parameters like that I'll generate a payload to be submitted via our HTTP request now one thing you'll see about one big difference between Python and JavaScript is the curly braces in Python represent a dictionary in JavaScript they're actually an object in Python the keys of a dictionary can be anything and JavaScript the default keys of an object or strings so this does not mean the variable F in the variable a it means the string F and the string a in fact we can even have a shorthand here where in a JavaScript object if you don't say what the key maps to this is an entry where the key is the string s and the value is the value of the variable F that's kind of nice so nice little shorthand and I'll copy this four times a b c ii that's a mistake number what are we up to 23 higher and then finally we'll say if size was provided payload size equals numeral size value now one nice thing about es si es 6 is normal javascript historically was not very forgiving of things like trailing commas es 6 at least makes it friendlier for us to use no originally javascript was just not as friendly as yeah we're almost done Ruby in just one or two seconds save it now will actually submit the request and you'll see that this looks very similar to you know our Python request library we do a put on some URL so I'll I'll give you the URL in a second we send the payload and then we establish a callback and that callback takes it as an error and a result the callback is just some function to be called once the request succeeds if it's an error we just return otherwise and here's the magic of this and this is the last important part of react and I'm showing you an credibly brief introduction to react I will get the results map and the pending map out of our current state I want to update that state with some new pending jobs what I'm going to do is as follows red stop body is the body of the result of the request now if we when we saw that we saw let's see if I can find HTTP GET of one m0 oh I need to do a put tonight so I'll put that and I'll get it back and you can see it gives you this result in this first request we need to pull that out of the body so I'll say Const result equals body and that's our ID and I'll do this timer equals set interval i'll create a timer with that result ID using the set interval function and what I'll do is all-pole my endpoint every second until the result comes back or every hat every half of a second until the result comes back and then I'll do this is the magic of react this dot set state I will set the state of my component which will trigger a rerender Inc and I'll do this state lessly so I'll create a new mapping so dot dot is our equivalent of the star star it's called a splat operator in JavaScript and Python it would be the keyword unpacking operator this syntax exists in Python what star star in Python 3 5 and above why am I telling you about Python we're using Java JavaScript programmers now and I'll just add this one element here yeah okay now notice I have a constant game result I need to rename this here's how you rename things and react see I'm almost done I want to at least take you to the and then what I'm gonna do is I'm going to I'm gonna do you know in the cooking show where they say where the cook just kind of throws and gether to make it so easy and they pick from the bottom oven where their staff had made it earlier perfectly that's the last step that we'll do here we have our little app and we can submit and when we submit it should show a little pending there I'm going to copy from my final react app just because we're out of time and I have to make some small changes to this give me a second because some things have changed in our things but essentially all we did was we added in that polling function and I think this should work okay there we go okay this is our final one and let's submit x squared zero zero zero two let me guys don't let me grab my other my two other ones here give me one second just to grab my okay cuz I'll show you some okay so if you can hold on one second oh and there's one really incredibly unfortunately annoying thing if only we had another 30 minutes I could show you I could show you live okay so we can finish this up tomorrow so the only pieces that we have missing are just to finish our react app and to actually add in the bouquet response now I will show you if you'd like to see what the end result looks like I'll show you what the end result looks like so give me a second to just kill these servers here here here and there and I'll show you the I'll show you the final there's my reddit server there's my celery worker there's my REST API and there's my that let me restart this is what the final one should look like and when we submit it submits a job and it gives you the results and if we make this really long one you'll see it's actually polling every half a second for this for the past to be complete this takes a long time to view because this is where I should restart this because that that's too many points for bouquet to show what the way down during the plot so there we go so you can see it's still polling it's still giving the result and 100,000 fill a lot if you give it a second while you're packing up you'll see it pop up the bouquet graph try it again with something a little bit smaller a thousand might be and so that that I hope you enjoyed that oh it came up just as we restarted there we go there we go and you can see it's penny for the result [Applause] [Music]

Channel: PyData

Views: 28,633

Rating: 4.747088 out of 5

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, download, learn, syntax, software, python 3, data scientist, data science, data analytics, analytics, coding, PyCon, Jupyter, Notebooks, bokeh, numpy, celery, react

Id: eEXKIp8h0T0

Length: 109min 1sec (6541 seconds)

Published: Sat May 06 2017