AWS hands-on: running serverless Django with Zappa

hello everyone it's 3 p.m. so Allah let me start our worship today greetings to everyone from from Tallinn well I'll be spending the next few days a beautiful city no training outside but I guess you know everybody in the Baltics is is used that kind of ever I'm my name is Jules I'm the CEO and co-owner of cloud wiser and we are hosting today's workshop on running serverless django with Zappa and I'm really happy that you spent you know decided to spend some time especially during the the summer time to get deeper into this topic and this worship is specifically dedicated to the local EWS community and this and then a number of our events were you know we're running this with the aim just to grow the competence of people using a double yes in in in Latvia and in Estonia and in lavinia and to also encourage everyone to share the knowledge which a good example will be today off of class and sharing his experience with running some specific stuff on on AWS and actually we you and have special slack channel for that for for the local AWS community and I really recommend everyone here to join it and I'll be sharing the link in the chat after after we start and yeah like local community wide where dustcloud Weiser care about it so you know well first of all where the local AWS partner in the Baltic states so we're definitely like open to to working with everyone this that's using or speaking about using double yes and on the other side like it obvious resell is our main business lines so like from that perspective you know the more you get out of AWS the more you can use it in your in your business technology so you know the better it is for us as well and on that side so but you know getting back to the workshop our host today like I said will be class from a very special in a nice startup from from Latvia in Oregon and I guess he'll be changed some details about himself in the intro but before we start I just know just a few logistics details for for us to like get started on the right foot so first of all the plan is to spend up to two hours on the workshop we'll see how it goes and how much time it will take maybe we'll finish a bit bit sooner if you know the demo gods will be with us this a Q&A session at the bottom of the zoom environment so please you know all the questions are very welcome and the plan is to answer them at the end of the presentation though of course if you have some some troubles just write them down in the chat and we'll do our best to help you out and we'll be recording the the workshop so this means that we'll be sharing the recording with everyone after after the event so you can go through it once again and hopefully you've seen in in the emails we've said that we have also training accounts already for those who would like to go the whole hands-on experience together with class so if you just need that you see the email in the slide so just if you decide to go with it just just drop us an email and we'll sell send the credentials right away so that's I guess it from our side cloths are you ready to take over okay class I think you're muted yes sorry about that zoom just as you stop sharing zoom decided to go off screen and lost the controls for it yes sir good afternoon everybody and let me just share the screen so I get the presentation going alright so without working oh okay awesome so right so short things about me like I'm the head of infrastructure at Nord again we're small well not that small and more vintage start up working in basically data analysis and enrichment for open banking I've been a DevOps engineer for as far as I remember which is at least the last five years and working with AWS for the same amount of time and I also happen to be a total geek and what the things that we are actually going to be touching upon its AWS lambda which is their server this implementation server less being in quotes since it they're still servers it's just that you're not managing them that's handled all the way through by AWS in comparison something like easy to where you still managed software say side of the hardware we're also going to be looking at supper which is kind of a framework that allows you to package and deploy existing Python applications into lambda so that means you don't actually have to change the code you have to initialize a PI in it and you can use it to do various functions within the lambda environment we're also going to be touching upon dine account which is diamond dynamic environment management package for Python and the reason for that is because it allows you to have really convenient external conflict management for example you can run have your configuration in Redis secret stored in vault and be like so there's no real login and environments so you're not reliant on an individual piece of tech yet you can still leverage the fact that you're running in lambda and you can use it together with the other AWS hosted tools such as readies so it's pretty convenient other thing is that's not here but that's more or less the main outline of what we're gonna be touching upon like we're not gonna be going all the way into each individual piece because that would probably take more than a couple of hours but what we're going to be doing today is basically a simple tap shop on each of those pieces and basically deploying an application in various stages of exec its existence on one de right so if everybody is ready then maybe we can actually start with that and maybe you can post in the chat if you need a couple of minutes setting up the environment still as the email wasn't out I believe that the expectation is that if you want to join in on the hands-on experience that you have basically the github repo I can actually post it and shut down loaded which ok that's refusing key open so I'll just do it from the other browser you and where do you get shot there you so assuming that everybody has that repo downloaded the first thing we're gonna do is basically just pull out it's the base folder and that's mostly what we're going to be working with the other ones are basically interest like snapshots of the various stages just in case they're later you need to look back at something or des like for examples but mainly we're just going to be using the first one so we're just gonna copy the first folder into basically anything as long as it doesn't have the numbers because the numbers mess a bit with the environment naming that's good and automatically by Zappa so let's just say workshop workshop if you look at the code and then it's actually fairly basic Django application it just has the config module that doesn't is practically just bear initialize the Django application and we have a simple pages module that has single view saying hello world so nothing complicated so that we can just keep it simple today and once we were in there the only thing we need to do is basically activate the PIP environment and installed the Django so let's start with that so with that you can just run and install Django we specify a bit of an older version simply because to get a later version so running you have to do a bit of fiddling it's nothing complicated but just so that we didn't bug down the workshop with that we keep it to an older version we run that and that automatically creates the pip environment and it's Django is a dependency it takes a short moment to get it up and running right and well that's not an angle we can look at God that just adds not leash I guess we'll add later the big file to the application right so it installs it we run the pen shell and it opens the virtual environment already right so we have a virtual environment with Django since it's a super basic application we can just test it run Python managed by and run server and we have the application running here it's just complaining the day we haven't run the migrations which would be something that's needed in an actual application but considering right now we are just running a basic hello world example and that's not really needed and we can check that it's running and yes but that's just the part where we get it running locally now to actually get it up and running in London all we have to do is run pip install Zappa you you which grabs a couple of dependencies and along with photo so it can actually interact with AWS that also takes a short moment you take it and if I hope nobody's having issues following this but if you are just posted in chat or if you need larger font size or something like that yeah so far the chat is pretty pretty quiet so hopefully everybody's doing good that should be the safer assumption or optimistic rather since it does have a fair few dependencies it does take a moment but the good news is that once that is done any kind of interactions are fairly quick and just we reiterate we also need to have your credentials like credentials installed already in AWS configuration so in case you haven't done that in there's still time easiest way to do it is just run AWS configure if you have the AWS CLE if not go into your home directory make their make dot AWS directory and in the create a config file with the credentials I'll not show that since I don't want to screen share credentials but if you need any help with that just posted okay so we have zappa installed and the highly complicated way of getting it up and running is just running Java in it which will ask you for some default configuration for the most part we can leave it to its defaults like first part us what kind of environments you need you can specify multiple ones for now let's stick to just the basic development one here it asks which profile you want to use that pools already from the AWS profile configuration so in this case it's the default one for me if you have a differently named profile one specified that and then it asked for what kind of bucket you wanna do use here you can specify specify the config like the bucket that will be used for file file storage which is used in the meantime and I see that somebody is asking for the template just second posted you you you I'll post it in chat class I see a question with from yeah I'm already answering that I'll just writing it up in charge you you for some reason it space now you might also want to add region there so say you West one ignore the whitespace but otherwise that should be enough and then you can leave it as the default config right so back to that right so the bucket isn't actually used that much so the name it isn't important unless you like to name it have the s3 buckets clean and easily understandable but you can also just leave it to the default the bucket is mainly used to have word that moment where Java packages the application uploads it to s3 and from death the lambda image is built and afterwards the bucket itself and content is removed so we can leave it the default one and then it the final question is basically asking for the Python projects configuration and modules can the file or specification and the zappa has some logic to detect the type of application it is so if it would be a flask one now finish that if it's a classic one it would detector if not then you can specify whatever because it also works with basic Python notifications as well and I see Ilya what do you not see there like ah right I did not answer I answered two panelists not everybody sorry you do you see it now yeah sorry about that yeah and as we can see in the case of Django it automatically automatically detects the right one and we can just use the default one as well and yeah here you can deploy it globally which for most cases unless it's some kind of high profile production application is not actually necessarily necessary and in this case definitely not necessarily since it drives up the costs of it and hence way also the default option is not a void globally right and what it does is create this simple configuration file that if you leave it and go look at code that opened here we can see already it creates this file and what it is is basically a simple definition of the various environments and the region is automatically pulled from your profile if it's defined if not it would ask for it and that's the profile name which again is the AWS profile the name of the project which is why we actually renamed folder because that's the default one where it pulls it from but yeah any of this can be renamed runtime is pulled from what you are the virtual environment you're running when executing the initialization and the bucket there are other things that can be specified here man will get on that later but for now we have the simple dev environment definition so let's try deploying it deploying it is also intricately complicated it's disapper deploy and then the environment which in this case is there so what it does now it creates the I am role serve that will be used by London and also for the purpose of creating all the surrounding things which the s3 bucket the API gateway and the lambda itself then it also packages the application which is collecting all the dependencies and uploads it now see it's fairly quick but now we will face a little problem in the sense that if we open it it will not actually run and that is simply because Django has the default built-in security measure that you actually need to specify what hosts are allowed but that's not a particularly big problem because it actually specifies that you need to add this to allowed hosts now we just copy the URL and go into the project directory and go to config settings and here you have the allowed house but and here you simply in single quote quote copying the aloud host save it and now we can also see how updating works which is Zappa update def what environment and the difference in update is that it doesn't actually touch the API gateway or anything it's simply repackages their application with made changes and uploads it and deploys the lambda which also takes a moment but here we are no basically the URL should have changed and if we refresh it boom we have a running Django application in Wanda now one of the nice things that you can also do with it is actually see the logs which is easy as Zappa tale and again environment a woman and you have and you can pull the cloud wash logs directly through the CLI so for instance previously if something is not working you can already find it in the logs directly you don't even have to open it well you'd have to open it for it to trigger it but you have easy access to the actual logs here which would be something you need if you have disabled the debug state for example and if I'm not mistaken this also actually is real time so while you're running it it keeps updating the logs as they come in so it's quite handy for debugging certain things and the tale function also has filter options so you can filter out for specific logs which is quit and they for example if you wanted to filter out for specifically HTTP related logs you can run those upper tail there you specify what kind of logs so and you get only the HTTP responses and not actually the content and that also works in revenge reverse by specifying on HTTP so it's not exactly something that you'll be using more most likely in production since well you should at that point you should have some kind of log management that sort of things but for simply debugging while playing around with dev environment or making some changes it's quite handy to have such easy access to redouble logs the SEPA support gradual Bluegreen cannery deployment models it supports it in the sense not inherently but you can work around it by deploying to having the separation of environments and having been some form of load balancer or additional logic unto it but directly doesn't handle it now you right so with that we actually got through supper and now on to the next part which will be adding the configuration management and getting it in the project is slightly more involved and then running a single command it's due but the installation process is the same you simply run deep install Dinah con now which I did on the wrong screen sorry about that right and for this initialization you actually have to specify which file you're like what settings file is used in your application for that we simply specified by exporting it as a variable exporting Django settings module and then in our case the settings module the application modules name is config and settings and once that's run we should be able to simply run dynacord in need dish this already detects the settings and there we run it and it already adds its hook basically and the way it looks is if we go in the project if you open the settings file if we go to the bottom what the dad's is basically these two lines along with the comments on both sides so what it does it basically kind of hijacks the the settings configuration and injects itself into juncos setting format so in the eyes of Jango nothing has changed but in the context of managing it you now have the manage not that the settings config settings the trouble but also their support for different formats depending how you install a how you install it so for instance we can also add additional config formats by installing the Dyna confer like extra packages for instance we can install Dinah Kong yamo which adds extra dependencies to dinah comes packaged to the project in the context of dine accounts simply for parsing llamo in this case but it also supports Python files there's various external ones as well it supports Redis and Walt and two separate formats of settings as well because you can use it also for secret management and and separately for config management so for instance in a production format you could use Redis for the configuration management so that it's completely decoupled from the application itself and Redis for some form of Secrets towards a hash code Walt for a secret storage where you can inject whatever kind of Secrets you would need like applet that going as far as for example a injecting the Django secret key which in this case are sharing because we're not really running it in production but that's fairly handy because one of the drawbacks of not managing your own system as is the case with lambdas you have to kind of work around that or you could use the default lambda environments but then you're kind of stuck in the lambda specific world if you're using something like this you can use your external configuration which can then be reused if you're for instance deploying to ec2 at the later time and your architecture doesn't change or you can mix and match depending on your needs or say just use lambda for like running certain functions of your overall bigger project now in our case the way we can test it is ad settings dot llamo file the way the the way their configuration is formatted is also environmental so you can set for the application itself what environment it is and based on that it will pull in various conflicts so you in this case we just stick to default and say add debug false and again here we around the zapper update you ever now it should package as well dinah convent a into the dependencies first I congested and slowly uploads it right well we won't actually be see missing that the bug state is broken disabled because we're not really breaking the package but we can do that again and and meanwhile at least I can check if the application is still running which it is so dependency should have been added now how do we check that the debug is actually broken let's just for instance take the page and add some broken request to it and we update the page you you huh and it failed mainly because it's a slightly too broken one but if we look at the logs it's actually failing to start completely because formatting is bad but which also shows that here we can actually see that the debug state is not enabled because if it would be we can test the difference so currently change it the true update you you yeah and here we can see that the debug gets enabled again and that's quite handy even working for with working well developing it because you have a clear format that's reusable across the board and you have the choice of using whatever kind of format and central location you want for the country itself and since you can use python or ginger any kind of that formatting you can have actual and proper logic in this dynamic settings as well and even templating based on various settings or environments between the deployments which can be quite useful especially if you're declaring generating various environments where you don't really want to manage the configuration it can be auto-generated and deployed that way so you're not even touching the settings menu only which can of course be achieved by the basic settings as well but this way at least you can centralize it and irrelevant of the Python kind of framework or whatever you're using have the same coherent sitting structure and then for the last part on how to actually set up different environments it's more or less as simple if we don't want to initialize then we can simply copy this remember to add koume change the environment name to say stage the rest of the settings mostly stay the same except and change the bucket say stage or actually even the bucket can stay the same because it handles that logic so for more complications like we could configure the environments here and have have the different settings and based on these environments already in the logic but we're not going to be doing that today and if we now run the Safa deploy stage you it already starts creating another API gateway another lambda role complete isolation from the previous role so in this case the only thing that's connecting them is basically the s3 buckets that they use for deployment which you can still separate if you need to you can also deploy with in the VP's fees will suffer but again it's a bit more complicated but it's the importance is that they're a little bit because they're really nothing about it is that complicated and it's really simple to use just like Dinah calm the only problem with Dinah calm fears if you already have exists existing configurations with some kind of complicated logic then it might be a bit more difficult transferring it over but it's still very possible and the benefits are there and at least in my opinion worth it and again now we have the separate environment you which will have the same issue as it before but it's not into not enabled there and a loud host because is it's a different environment it has a different host name and regarding the host names they can also be changed and supper actually supports automatic certificate generation if you have configured domains you it supports both the AWS certain ensure certificates and also let's encrypt ones and it handles that almost automatically you just for certain mnsure context you have to have the domain enabled and I think it's North Virginia one region because that's the one that majority of AWS services for different environments you can specify in whichever config language that you're using you can specify a dev debug true stage debug false and whatever else and then depending if it's some specific environment it will first pull in the default ones and then if there's anything environment specific then it will use those to override or set the new variables you and that for the most part is it from the actual technical stuff we can move on to they're pinning use cases like the way I see the use case main use cases are the fact that you can quickly get it up like as you so the amount of work needed to get up the environment is not very long you can get that existing project for the most part fairly quickly it has Zappa specifically has some issues in that it depends on prepackaged packages because lambda out that the box does not support everything so you have to build some certain libraries which in the case of supplement for the majority part is actually already done but there are some exemptions like the reason we are using Django 2 in is because the SQLite package that they're using is somewhat outdated but it's not super complicated to both build and package the library yourself and then you can know like all of its in github and you can see how the logic how it does that and work around it or use Postgres or which is actually another thing yet you can use different database connections as well the only caveat to that is that the more kind of external dependencies you're using the more latency you're gonna have because the way lambda works is that for the most part it's dead well not in use so what Zappa does already is it creates also keep alive functions or keep warm in the case of lambda so that every four minutes calls the lambda so that it's considered warm and active which by default lowers the latency to it and so back to the use cases it can be also used for quick demos scalable applications that are not super intensively used constantly if you have applications that are constantly used I would say it's a better choice to go with something like ec2 or elastic beanstalk unless you are very very stressed for development resources mainly because lambda can get expensive where if you have very intense constant usage because as it scales a single lambda instance use country runs one connection and if more pair connections come up well that's well ones already running it spins up another island and that can add up if you have constant pressure but if you have the case where you can sometimes at peak times just get sudden bursts of requests then it's still usually well up to a certain point can be cheaper than running dedicated ec2 instances and you don't really have to worry about the scaling that much up to a point because yeah I think out of the box lambda has the scaling limits do I think a thousand instant instances at the same time and if you need more you need to request the AWS to extend that limit also another thing use case that you can run with Zappa you can set up also scheduling where basically you create the lambda used it creates the lambda for you and you specify a function to use within your application and then you can schedule the lambda to run at specific times to execute only that one specific function for that that can be used for example to you do some kind of database migrations exporting data from your application truncating data pretty much anything that you would need a cron for and in this case it's grown in an environment that once it's running you don't have to touch it because it will just keep on running and it's fairly good for like quick starting good these kind of projects since again if you're running this format you're not actually writing lambda code you're making a Django classic what they were Native bison application that can run on alum this so you can in early phases just run on lambda and once you grow swap to ec2 with practically no issues looking at your question Reynes yeah with the different settings you can pretty much change all the configuration so you have environment one environment Ron uses SQLite it has the dev config and all the debug enabled test two is using pause graphs or a MySQL database so you can run different Norman's like different back-end configurations without any kind of issues and for config as well you can have environment one using files or local environment variables or and the second environment using ready says the configuration source so for use cases I guess that's the main part and actually we've gone through fairly quickly and I guess we can officially go to the Q&A direction so if you have more questions then feel free to bring it up yeah I saw that the Q&A once went pretty well in the chat as well yeah but like if anybody has more use here so anything you know use the occasion and then should them I'm sure cops will be happy to answer all of those not only like specific technical questions but maybe some use cases or scenarios like like rain is also nice see that we have the nail here in the in direct inspiration to all of this as such while we were waiting for the questions I can also add that yeah like I think it's very nice that the community event is being hosted by the members of the community so if anybody here would like to take the same role as classes doing today or the nail was was doing a few weeks and also just let me know and I'm sure we can set this up and I don't know cause maybe you would you most probably would say the same that like any like knowledge sharing or workshops from from from companies and people you you know or that shared this similar view or background it's it's way more interesting than just I don't know you know official AWS without the doubt because like in these cases like this is not marketing for anything this is the tech that we find interesting and useful so there is no ulterior motives to this like specifically for this tech yeah the only reason is really because I think it's actually kind of cool and it's the lambda combination is kind of green fringe ish a use case for Django because that's not usually the way to go but I find it as an interesting approach that can come in handy in certain cases no questions coming so maybe we can start the wrap up of the worshipers anything like from from the agenda Oh like how far have you gotten or like depending on how how many people are interested we can if you're the only one we can do one one on one afterwards as well but specifically for the settings you run you initialize dine account and what might have been missed is that I missed myself in doing is right that poster sharing changes country right okay so first installed the Dinoco what I missed was that you actually need to install the llamo one which is specified through the extension formula like that that installs also the support for Yano then we need to export the specification of where what is the location of your conflict settings which is I can copy-paste it in chat and then you need to run dinah conf in it in the root of the project folder so that will be default a folder where you have your managed by and in this case in my case it will complain that it already exists but that's not a problem just might you might want to check in your contract if it actually updated the settings by should be on the last line you should have this nobody else has any questions that's not a good job long the way like yeah if you don't we can contact afterwards and talk one-on-one with Ilya Ilya are you on the slack community slack if you're on the communities like you can just write me up and we can go through it together probably easier to do it okay so it will be like a good excuse for ya - John yeah and then cross is there and then you can follow up on haha got some sneaky marketing be enough so yeah first of all I guess I guess both from my sight and and then of all the Erinyes thanks for preparing this and I know it's it's not that easy to to do such things especially you know the demo gods are not always that favorable a moment you know even have you know running this for an hour is really much probably to wait a more time than now did you see all the all the options so yeah thanks once again and as I mentioned before to everybody else if you want to to follow clouds example just reach out to me and I'll leave my contacts at the very end and I'm sure we can work something out for you to also present and show something to the local community a class was very good at reminding of the slack channel that's already covered and before we close you know feedback is very important I guess everybody that helped helped out in preparing this workshop so I'll be just appalled so please just just give you a feedback on how you like the the workshop and that would be very useful for us on on preparing for the other work and also like in this fact channel or just in the follow-up email that that will be sending out you know if you have any ideas or topics that should be covered in similar events in the future just let us know and you know it's it way easier for us to decide what to do next if we get some feedback and ideas from you okay I see you only two people voted now come on guys and that's well there's only seven people here together I'm not counting that okay that's that's wait a few moments more okay but still [Music] the feedback is pretty clear I'll just be sharing the result yeah so I think it went well the feedback is also quite good so looking forward to seeing everybody here once again any farewell words from your class thanks for the opportunity for this and like yeah you already mentioned like if anybody else ideas look maybe you'd like to see something more in depth or some kind of specific use case reach out let's keep in touch in the slack as well on everyday issues I guess there should be some interesting stuff we just need to get started as a community like in the other countries and you know there's plenty of stuff around database that can be discussed so have a nice rest of the evening we saved almost one hour of your time your book let's keep in touch and have a great summer everybody one all right bye guys you you

Channel: Cloudvisor - advanced AWS partner in the Baltics

Views: 697

Keywords:

Id: oRaUXvUWJ0U

Length: 65min 52sec (3952 seconds)

Published: Wed Jul 08 2020