Benny Bauer Python in The Serverless Era PyCon 2017

hello everybody and welcome again back to our final track on the pike on 2017 - last day Sunday thanks so much for being here on a Sunday afternoon you're going to be excited I think we're going to have a demo from Benny Bower who's going to talk about Python in the serverless area please do me a favor do all of us a favor by turning off your electronics or at least making him quiet and we will have a 10-minute break following the talk so there's no need to rush out to try to get to your next talk will give you plenty of time for that everybody please welcome Benny Bower hi good afternoon am I on okay so glad to see you minute for the third day I wasn't sure around menu will pop here so in this session I would like to talk about build house which frameworks there are to build fight and surveillance applications I'll start with I'll skim through the what is survey list and we'll get to the demos as fast as I can my name is Benny Bower I came all the way from Israel to talk about service and enjoy Python I also heard that Trump is coming for a visit so I said it's a good time to go so here I am I'm a chief architect that discovered a brand new startup our mission is to tell the story of design items and the places they're residing and the creators behind them prior to that I was a cloud architect that's AutoCAD in the Autodesk for seven years so let's get it going so how many of you heard about server listens for I guess most of you how many are using server lists okay so serverless is the latest evolution of cloud development you can see kuvira's and another abstraction layer on top of compute and platform and infrastructure resources which makes it easier for us as developers to build applications and ship them but service is also an overused term and I'll try to to make some order in it so there are three main meanings to surrealists the first one which I call developer success or software is a service which is designated many service managed services that deal with specific things in the usually provided by third parties for example if you need authentication for for your app you can use up zero Tullio forest and that's ending al-awliya for search and you get all of these out of the box so you don't need to implement the backend of them another another meaning for service is back end as a service which is things like firebase and AWS mobile app they're basically a set of common services that you will need to build your mobile application for example you probably need authentication you need some kind of basic data store push notifications if this is all you need from for your mobile application you probably want to use one of the back and back into the service services instead of implementing it by yourself and maintaining a back-end for this and but there are still times maybe most of the times when you have your own business logic that you need to run on a back-end and this is the third meaning of server lists which is TAS or function as a service and they're most of the cloud providers or the largest cloud providers have a service for this o AWS has lambda as your has function do thousands cloud functions IBM the bluemix has opened whisk which is also open source and this is so last year when someone mentioned server lists probably me mentioned SAS ok and and this will be the focus of my talk here so what what exactly is says so it's a fully managed compute you get out of the box a provisioning of an instance you get the patching of it you get some some basic monitoring and logging you get scaling out of the blocks and which means you have less operations for provisioning instances or maintaining your your server you take your code and just deploy it package and deploy it and then the nicest thing about it is that you pay for the actual usage and there is a resolution of hundreds of milliseconds which is I think that's the thing that made server lists or function as a service that made a hype out of out of function of the service and basically it means you have no idle servers no idle compute so how it works as I mentioned you deploy your code but then you need to define triggers to what will make this code execute and triggers are many types of events that you can configure it could be just in HTTP requests so you define an endpoint that will trigger your function it could be an event of uploading a file to the storage service of the provider that you're using it could be a change in a in a database row or a new row in a database that is provided as a service by that provider for example dynamo to be in AWS and there are many other events have scheduled tasks and many many others and there are new almost every month you can hear about a new integration with another service in in debt for that cloud provider and and it means that the architecture of the application becomes event-driven okay and and this is this is a big change for building cloud applications if you are used to build applications that poll to see if something happened so now it's the other way around and then when the event occurs it triggers the specific function that you configured and the code is being executed and you don't need to worry about all scaling and redundancy you get it out of the box and the use cases or analyst these are just a few within we can have talked only about use cases but the main things that you will hear about them is building a back-end so it could be a web deck and mobile backend box or another popular use case is a data data processing using those events of when files uploaded when something is changing the database when when you have a data stream that you want to to process this is this is very common so as I said analysts they're so many so many use cases and every other day there's a new post about some about new use case but there are also things that you need to be aware of when when using function in the service or service architecture and I'm not going to go through them because again it's another another topic for a full talk but you need to understand that server list is not a silver bullet unity understand it if it works for your use cases and you need to know the limitations and how to overcome them ok so we got to buy some okay we're in baikin so let's talk about I can and functional deserve so out of the cloud providers AWS IBM open whisk and Microsoft Azure support fight on both to Python 2.7 and 3.6 Azure is still experimental but you can already try it but if you already had a chance to try and Detroit functions and configure everything you know that it's not easy and there's a lot of things that you need to tweak and configure and define and package and deploy and we as developers we're lazy and will always invent things to make our life better so their frameworks for doing this stuff and these are some I think most of the frameworks that support Python runtime will go through some of them I'll start with serverless so if the term service wasn't confusing enough it has another meaning which is a framework called service by the way all all of these frameworks are open source and then you can go for the code you can contribute you can add stuff so the service framework think it's the most mature and when I'm saying most mature it's a 1 1/2 years old maybe surrealist is is such a new thing so 1/2 years in the cereal stands is a lot that's that's what makes it the most mature and also support the different cloud providers and various runtimes so I want to show a demo of building some simple Python application with the service framework surrealists frameworks itself is built in no js' you you installed by using NPM install serverless and once you installed it you can create a project you can create a new project out of the templates using the CLI so this is something that common to all the frameworks they usually the way they work the you configure you use the CLI and you can figure a configuration file with the rest of your code and the framework does everything for you so I want to create a Python project for AWS to the 4/3 AWS so they have templates so my minus t4 template on have Python this one is is a Python 2.7 and let's see what I got so sorry my dad I wanted to create a new menu folder okay so what I got is I get two Falls created for me I got the circle this Yano which is the configuration file and I got a template of the function the function itself that I'm going to deploy so let's start with the service Yano from the configuration file you can see that you can configure the provider as I said it's not only for AWS but this project is the template for of AWS and the runtime is Python 2.7 and I want to configure that the region will be Oregon so it's yes West - there are many things examples for many things that you can do that you can configure but I'm going to skip them and I'm getting to the function itself so this is the definition of the configuration of the function so I have out of the template they created me hello function in the handler PI file and now I want to bind an event to trigger this function so they have some samples here I'm just comment out the things that I want so I want to create a simple HTTP API that when someone calls hello and provides a name argument and this call is a HTTP call it will trigger my function and let's go to my function so this function was created out of templates let me just move some redundant things from here and what you can see in this function that I get two arguments the first argument is event event is the input of the input of the event that triggered this dysfunction and context is the runtime information about the the specific lambda so I have a body that I want to return and I define the response the status code of the response and and the body that I have up above here and I just want to say hi and I want to get the name out of the argument that was sent to me so this argument is in the test parameters and as we configured in the llamo file name is name and that's about it now I'm ready to deploy all I need to do is SLS deploy and hope for the best so what happens now the frameworks the framework packages all my code and start starting to create the resources that I need on AWS to be able to deploy the function to bind it to an API gateway which is the HTTP gateway that map's an endpoint to my function it also configures a bucket to upload the codes to it will configure the different security rules that I need to - for the API gateway to call the lambda function and also log everything that you see here if I if I wasn't using the framework I needed to do it by ourselves either by going to the AWS console or using the AWS CLI when here I did it with just by configuring a configuration file and using the CLI to deploy it so it usually it takes a few minutes it also depends on the amount of resources that you can figure on for the for this project hopefully it will be done maybe in the meantime I can show you what other things you can you can define in the Yama file so I could have defined that the event that will trigger the function would be a ns3 event which is the file Apple Store the file Apple storage event or a scheduled task or many other different things that can trigger my my function I can also pass environment variables through the configuration file I can define other resources to be created on AWS if I need let's say I want to create another bucket for my project or a DynamoDB table so I can do everything from here okay so it's done I have this endpoint let's curl it and say hello vikon and I got it I got the response back then I need to spin a server then it needs to do anything to go to a console everything is here from the CLI okay but that's the simple case now I want to add some dependencies to my Python project so I'll create a virtual environment and just update my tip butter us okay and let's say I want to add arrow because I want to display the time I know I can do it with day time but I love arrow so let's do it with let's add arrow and I also want to create the requirements file okay so now theoretically I can go to my function doing for an arrow that's some time rival that's right a time and let's write both the you to seein and the important time just to justify my my arrow so it's - yes cific okay but how will how will serve a list or now to package online requirements and also bind them in the code so for this the nice thing that you can do with with server list is that it's pluggable so anyone can write a plug-in that hooks into events in the framework and implement stuff so there is a nice nice plugin called service Python requirements I will still it as I said it's it's a node so and I also I also need to edit to my Yano file to configure it to configure it to work with it so let's go to the ya know I led up plugins section and put it here and again I'm deploying and now you'll see that it takes the Python packages from the spiritual environment and it will create the requirements folder will my packages and package everything together deploy to to AWS this time it only goes for the changes that are needed to be done so this deployment should be quicker and let's curl again and I get all the data time and the the message so this was an example of using circle this framework let's see another example that is unique for for Python and this is a friend called Zappa it was developed by rich Jones and the idea of Zappa is that if you already have a whiskey server for example you have Slough or jungle and you want to take it as a whole and deploy to lambda because you don't want to manage your servers so you can use that for this and it's a super simple the way it works it it's packages your let's say Django or flask application it uploads it to a lambda function creates a proxy from an API gateway and so you don't need to define every endpoint that you have on your API and it handles it for you and it's super easy because let's say I have here a basic flask application with two endpoints it converts currency so if what I need to do is just create a Zepa in it I already pip install SAP ok and when you do that by in it it will just ask you for different configurations I'm going with the default here and it created a configuration file which call data settings so again here I want to change the AWS region and there are many many other stuff that I can configure here but will not get into it but it has a lot of functionality that you can do here and again Zepa deploy it takes about a minute and it does again it happens it takes the environment environment invar the virtual environment that you're in it takes all the dependencies from this environment packages everything with your code and creates the needed resources on AWS the EPI gateway the s3 bucket upload everything creates a lambda and then you got it okay so let's take this and so we said that we have two endpoints so this is the first one the hello and we have another that conferred let's say from some dollars to euros and convert 20 oops to euros and convert 20 dollars and you got the response again it didn't need to spin anything doing everything from CLI with a configuration there there are pretty unique features in zappa for example you can do a global deployment so instead of deploying a lambda to one region you can deploy to all the regions in AWS there's also a functionality for keeping it's warm it's something one of the limitations of lambda where the way it's implemented is it runs on container and when you're not calling this container for five minutes it will become cold so next time when you call it it will mean might need two seconds to get warmer so if latency is something important you get here a kind of a trick of keeping keeping it's warm it also supports using the lambda packages and many Linux Wheels packages which are pre compiled packages for Python precompile packages that are suitable for the AWS lambda in since the linux version that they use another framework is a chalice it's by AWS it looks very similar to flask I like that but it creates automatically creates a separate endpoint for every route video you will define in your code but again it's very simple to use you create a new project from CLI and you define everything in your code and then you deploy last thing that I want to show you is is it's not a front iframe abut a library called fibrin it was developed by Eric Jonas he's a postdoc in Berkeley and it's simply a library that enables you to take a function from your code and at runtime distributed to endless amount of lambda functions so there's a nice example by Sean Smith that he used it for web scraping it created the that's the whole code so he has a script function that takes a URL and using pyrin you create a pyrin executor and then you pass equal to the max rank function of pyrin and you pass the function that you want to be executed distributed li and you pass an array of inputs so in this this example array of URLs that he wants he wanted to describe and that's it and what happens behind the scenes is that during execution pyrin will create those lambdas will deployed and package deploy them and create to configure all the roles that you need and basically it's just one line of codes and I think this is this is an amazing example of what you can simply do with with serverless okay so we got to the end if you take ways that that you might take is that serious is pretty fast to develop it's cost effective for most used cases within talked about we didn't talk enough about limitation but you should be again should be aware of when to use it when not and you can do it easily with Python and using the frameworks and for for those of you contributes to frameworks or build like to build stuff and open source think there's a plenty of opportunities in the server list domain because there are many problems and challenges to be solved there and there are many tools that can be built to solve these issues and Python is one of the best tools to do that that's it thank you [Applause] you have any question so we got about one minute happy to take one question here and then Benny's offered to also meet people out in the hallway to to continue questions yeah okay great go ahead hi um I really enjoyed your talk I was wondering if you have any preference or comments about the the the config files being like general or JSON format like on the ease of maintenance for me ya know is easier easier that's less code PMO okay how easy is it to switch between some of these frameworks because I know they're all kind of young and so if we pick one and then in three or four years it turns out we picked the wrong one music are the configurations kind of similar where you can obviously it's not trivial but like what'swhat's the effort involved have you converted projects between these different frameworks yeah so free or four years in terms of service is eternity so it's hard to say what it will be in three or four years but I think it's every framework has its own configuration and configuration style so it's not reveal the code itself is the same programming model so it's easy but take the configuration and and replace it it's not trivial but it's one file so it shouldn't be too hard thank you very much again thank you very much and we'll take a 10-minute break

Channel: PyCon 2017

Views: 8,424

Keywords:

Id: G17E4Muylis

Length: 30min 47sec (1847 seconds)

Published: Sun May 21 2017