Local Development for Serverless is an Anti-pattern - Gareth McCumskey

all right so obviously i'm here today to talk about local development in serverless um my name is gareth mckenzie in case you didn't catch the intro i'm a customer success engineer at serverless inc and basically my job is to talk to people about serverless and help them find solutions to issues they may have helped them use serverless tools serverless framework and so on um so yeah obviously as with all these talks uh everything we talk about is due to our own personal experiences over the years we've been doing what we do um just like my talkers as well and yeah uh taste timing this talk over the last couple of weeks has been a bit tricky for me it's i've always timed it out about 45 minutes and i know we don't have that much time so i've been able to strip it down a bit but we kind of need to get going so i'm sorry if i speak a bit quickly at times there's just a lot to cover so let's get going then to start off with what i want to do is sort of set the scene with local development and what we want to mean by local development what it where it's come from and why it exists why do we do this thing called local development in the first place and really it comes down to the fact that we want to have our machine act like production so that we can test our code i need to write this code i need to test this code and i need some way to make sure that it works the problem is is that traditionally in the past it's just been too expensive for every developer on a team to have an exact copy of production uh sitting around just for them to use that would be the perfect ideal is every developer had an exact copy of a production system right in front of them to to work with but traditionally you can't have a huge cluster of machines load balanced with mysql databases and a master slave set up and all this complex infrastructure that production systems require before serverless so one way we've and also obviously it's too risky to make these changes in production if you can't have production systems to use yourself the other possible solution is to just do it in production but that gets a little bit risky i've been there i've done that and it doesn't work very well so what we end up doing as a way to help solve this problem is we try to as best as possible copy the production environment on our local machines for local development and this is to solve all the problems i've said it's it reduces the cost and it makes us that we can actually test this code to some degree and we can then even have a and this is very common as well in a lot of teams is that not only does each developer have some kind of local environment to play around with they also they usually even have a development server of some kind some kind of integration or staging system that you can take all your code and all the other members of your team stick it onto the single machine that is even closer to what production system looks like to test with you know i know any problems and so on before you actually push us out into a production system and this is where we often come up with the uh here the mantra it's become a meme in the development space that it worked on my machine i've developed things locally it all works fine i throw it over the wall at the ops team and it breaks not my problem it worked on my machine and because of our ability to try to get things to run locally we were then able to go back and say well we're doing this web stuff that we would normally have to do on a web server on our local machines what can we now use from other developers that have built things locally like desktop developers and debug tools for example where you can step through code and break points and so on is a pretty nifty feature to add on top of this just another advantage to being able to work locally we've then gone ahead and to try and get ourselves even closer to production systems use tools like vagrant and vms and so on to just replicate those production systems and obviously today docker kind of has taken over that almost entirely and development teams are now you know replicating production systems using docker containers even in the serverless world so just to sum up about what local development means and where it's come from is that essentially production is just too complex or expensive for us to replicate so we've tried really hard with virtualization to copy all of this production stuff on our local machines and it also allows us to get a few additional optimizations to work locally but it still requires a step between local and production so no matter how hard we've tried to do local we often have a development staging system between our locals and production anyway and that's kind of what local looks like even before serverless and it's kind of become a industry standard i've joined i've been part of and and joined so many teams we're essentially either you joining a team and working with everybody involved to try to find ways to build a great local development experience for your team or join the team where you've spent a day a week maybe longer sometimes depending on the complexities to how to to get that simulated development environment running on your own computer so that you can start contributing to the team and this has become a necessity this isn't something that uh you know was created in isolation and created burden it was an absolute necessity prior to serverless to have these kinds of local development systems because of all the problems i mentioned before we have that we really have no choice they have helped us to develop foster and create things and eventually when we want to push out production most of the work is done but there is going to still be integration work to be done but at least local development environments in in those situations have actually helped us but how does serverless come along and change some of this so that's the interesting thing now we're moving into the serverless world we're no longer bound by the same rules first of all i tend to look at this from a web application development uh scenario because that's kind of what i've done most in the past so that's why my example is a bit sort of web app heavy but that kind of works with any uh use case that you might think of so first of all the commonality between local systems and production is often the web server you can have an apache or an nginx web server on your production machines and that's pretty easy to replicate locally but with a serverless world we don't really have a web server that we can just hook into and understand how it works and copy it in local and production we're going to be using a tool like api gateway which is more than just a web server sure it receives http requests but it handles things differently than a regular web server would and potentially adds additional features on top some of those would be things like routing this is an example of a laravel root file in php and i didn't know i was going to get so much php stuff happening in the conference today but uh here we go and this is a relatively simple one actually this is one of the examples i kind of pulled off of google images very very simple example of a laravel roots file but this is the kind of stuff that traditionally you'd have to write maintain in your code itself it becomes a part of your of the compute of your application it's not just part of the apache infrastructure that you're going to spin out and this means you've got additional stuff to maintain you have your frameworks that you need to update and maintain you've got more code to edit so now something like api gateway takes this work away from us we don't need to manage routing as code inside our application api gateway gives us a way to manage that as part of configuration instead and this continues uh with things like databases this is an example of a entity class for a doctrine orm inside a symphony and again this is very simple example and these get hellishly complex i know from experience but the idea with this as well is is that now i have to maintain my data models and my entity relationships in the code i'm writing i have to produce this code have it automatically generated in some way have a framework to maintain and deploy and implement and so on so this adds complexity into my compute layer instead of being part of the infrastructure layer that configuration with serverless tends to give us and it goes even further this is a cron job what about cron jobs these are those horrible things that you kind of think of think of later that you need to implement in your infrastructure usually ends up being a t2 micro instance sitting in an aws cluster somewhere on its own trying to do cron jobs again this is something we can just do with serverless now it just becomes part of our infrastructure to to configure our cron jobs we don't have to worry about implementing the stuff and maintaining it ourselves and an example of what i mean is if i want to set up some routes with api gateway this is obviously in the serverless framework but other serverless tools work very similarly where you're configuring your endpoints and your paths directly in the configuration of the resource that's going to be receiving the http requests so you're not managing this in code this is not something that you're going to go update and maintain later you might have to change the configuration for different paths but that's the extent of the work setting up databases can be just as simple with dynamodb now i have essentially a key value store that has no fixed schema to it so i can store the data i need to the only complexity of dynamic db well i say only is the querying aspect and there's huge amounts of information about that we've even had some of the speakers mentioning dynamodb but what's cool about dynamite db is it helps me get up and running with a database or some way to store data really quickly and easily and i can manage that through configuration primarily i don't have to spend time and effort with orms and entity classes and so on and it runs the gamut there are multitudes of aws services that end up lifting out a lot of the code that we would normally write to do things like sns for example which is a pub sub service so no longer do i need to worry about trying to orchestrate pub sub with a framework or a library i've installed it's a service i consume aws sdk push the data to edit and it does pops up for me i don't need to think about it uh sqs similar thing i can just push stuff into a message queue i don't need to maintain a library that manages messages queues for me i can just push stuff into the using the sdk there's step functions as well i state machines used to be a process of finding the right framework or library to include into your application and configure it in a certain way set things up so that you could have a state machine running in a specific way now i have state machine machines as a service all i need to worry about is the actual code that runs in the states themselves i don't need to maintain all the additional work around managing state machines and cloudwatch we talked about chrome jobs before so there we go i've got cloudwatch now cloudwatch has the capability to have scheduled events triggered and that's what manages our crown jobs for us i don't need to have that tiny little t2 micro instance that everybody forgets about that's running our cron jobs for us so what does that mean i'm putting all the stuff together talking about these services that consume all of our code and what it means for us is that we end up in the long term with serverless having less dependence on our own code previously as a show we'd have to have we'd have to include lots of code to run things that are now taken over by services because of the reduction of code apparently the more code you add the more chance you have more chance you have to have bugs apparently bugs only happen when you add code so if we aren't less of it that means we've got less chance of having bugs that's a very nice benefit to have by having code removed by services it also means that uh now that i have less code that also means i have less code that i actually need to worry about executing my own machine a lot of these services just do that for me so there's there's less for me to actually test now and the other thing that often slips by is that cloud services have now become integral to our application we're no longer writing code that i can take from one cpu and dump it on another cpu and it'll run because that's the extent of the the infrastructure that we've coded with it the cloud service has become integral to it and we have to have them available to us you can't just lift these out without the cloud services around them and and use them anymore you need them so we've looked at sort of what local development means where it's come from and looked at what serverless can what serverless has added to our traditional development in helping making testing local easier to some degree and i wanted to go through a lot of the current sort of methods that i've seen that i've used myself and that i've seen other folks using as well to try and do testing of serverless applications on on a local machine how can i execute or test my my serverless stuff here and now and not have to worry about using the cloud so the easiest one that i can think of is with the service framing just to run the serverless invoke local command it's the quickest and easiest way just to execute any old code sitting in a lambda function that you've coded on your local machine you run the command servers invoke local dash f in the function name and it's literally just going to execute the code it's not without its issues because access to aws services becomes tricky obviously if cloud services are integral to your serverless application you're probably consuming dynamodb s3 sns sqs and many others inside your lambda function itself you're calling out to aws so the serverless invoke local command it's not really going to work for that it becomes a bit tricky to configure and set that up so that might not be the best solution to that also unfortunately it does not accurately emulate the conditions of the lambda environment it doesn't quite emulate what the environment is like when a laminar function actually executes so let's just move on to the next one this is a very very common um use case i've seen as well with serverless offline serverless offline is a relatively simple to implement plugin into the serverless framework and really what it does is it gives you the ability to support http services in your uh serverless frame and your service framework service um but that's only that's really all it does if all you're building is an a is a bunch of api endpoints well you might be okay then uh this might be useful to you but from experience i've seen most teams uh including myself i've you know start out with an api because that seems that seems a very obvious use case it's very powerful for that let's build that thing but now i realize hmm i want to support image uploads and that involves an s3 event now if i include an s3 event similar offline no longer helps me test that that becomes a lot trickier to test now sns i want to do some sms stuff but that's also tricky now because serverless offline doesn't quite support that there's no support for the aws events and for local testing you also will need something like local stack which i'm going to take a look at in a second because you need some way to have these aws services that are being called inside your lambda functions again to execute in some way you can't just have your lambda functions error out because they can't connect to your aws account so you need some way to be able to run these services at least and it also it does not accurately emulate the conditions of the lambda environment that the lambda function be executing because it's your local machine it's not actually lambda so about that emulation how do we do this i mean i i have these services sitting in aws i want to use them in my lambda function but if i really want to test locally i need to find some way my lambda functions don't just error out every time i execute them locally because they can't connect to aws so one method is to emulate these services and this is usually done with a really powerful piece of software called local stack and there's even as you can see a plugin for the serverless framework to help you use local stack and really what local stack is if you haven't heard of it is a package that essentially runs either versions of aws services that have local variants provided by aws or close equivalence to those services to kind of emulate them as best as possible unfortunately local stack doesn't come without its own issues one of them being that it can get a bit complex to configure at times and this is actually a relatively simple configuration file that i was able to find uh where somebody's trying to set up those proxies and redirect to the local stack equivalents of aws services it's not absolutely monstrous so you know it's not terrible but you know you do have some complexity that you need to manage here now unfortunately as well now that you're using local stack you've got a bunch of docker containers including the services running within them that you need to manage as a part of running local stack and the local stack team works very hard to make this very easy to use and it's also not me ragging on the team at local stack this is an indication of the issues that happen with local development in general in my past i've worked as i said i worked with teams where we try to build local development environments but they become long-standing difficult things to maintain over time that you have to spend a lot of time and effort building maintaining improving as production changes you need to add more stuff to it and update everybody and that's and and you'll see that effort uh put into managing a local stack as well as part of your development environment it can get tricky to keep it maintained updated and running over time for all developers in your team and can cause a few issues and delays there not to mention that you've now got a whole bunch of processes sometimes consuming a lot of resources on your developers machines so it might not be the best for that as well especially if you're trying to run on lean little machines in your development team this can consume a lot of resources unfortunately it doesn't stop there [Music] unfortunately while the team with local stack works as hard as they can to make what they provide as close to aws as possible it's not quite there local stack is just not equivalent to aws and that's that's true of any local development environment it's never going to be as good as the production environment and local stack isn't any different um you also end up in a situation where you have you have you have choice to make between two choices you realize that you have a service in aws that is absolutely awesome if you can use the service it's going to completely change the way your app performs it's going to make things better it's going to make your life easier as a developer and you want to use it yesterday but local stack doesn't support it and this has happened to me a couple of times as well you decide you need to use this service in aws but you can't use it in local stack because it's not supported yet i'm sure we saw with some of the other talks aws has a whole bunch of services they've got a ton of stuff for you to use so the local stack team is still trying to add more and more of these services from aws into the package and they're just not there with everything yet it's just it's a moving target there's always new stuff happening so it's more than likely going to happen at some point there's going to be a service you want that you can't use in local stack so the decision is you can you can decide just not to use that service so your app is going to suffer for not using something that would make it better or the alternative use it anyway the problem with that is that now you essentially broken your local development environment because your tool that you've been using for that no longer doesn't support that service so you caught between a rock and a hard place do you want to keep your app less optimal or do you want to break local development it's a tough decision to make and also just executing that code that you've written for your lambda functions the the environment that it's executing in doesn't quite emulate the lambda environment very accurately so when i um in my first team where i started building back in 2016 i was part of a team uh trying to transition away from a php monolith to a serverless uh serverless microservices architecture sound familiar there's been a talk about it um and we ended up going through a process of trying to find a way to develop locally and went through all the steps i've been through right now to the point where we were looking at mocking these those services and i i i was a advocate for mocking aws services for a very long time even wrote a blog post for serverless about how to do this and why it's awesome um one of the reasons is it's a very easy way to set up uh testing for aws services there's a really cool npm module called aws sdk mock that you can essentially use and it just captures all requests to the aws sdk and lets you insert your own response to it so instead of going to the cloud it stays local and you can respond whatever you want a failure message a success message whatever it is from that aws service it also requires no complex configuration there's really nothing complex about getting mocking setting up uh in your server.jumble file you probably add another three lines to the configuration however and there's also no complex installation of applications so it's not like local stack where you've got this huge pile of apps that need to be spun up in the background or a massive development environment that every new developer has got to get somehow out of the code repo and spin up on the local machine and try to execute it it has none of that it's incredibly light and easy to run because it only consumes resources when you're actually executing uh your lambda functions locally and there's no maintenance there isn't a bunch of stuff that needs to be upgraded over time and every developer has to follow suit and falls and updates and so on very low maintenance and also doesn't limit the services it just ties into the aws sdk if it's in the sdk you can use this aws sdk mock tool to mock it it's not specific however there's some downsides um i'm not going to talk about the usual downsides most folks talk about with mocks one of the big ones for me is that mocks a lack local stack are just not equivalent to aws and this is briefly mentioned in some other talks as well while you can mock the services out and keep things local your responses will probably not be what aws would give you and even if they are you even if you're able to accurately produce the same responses that aws does this is not sitting in a data center uh communicating over networks you don't have the same network latencies involved you don't have the same im interactions involved there's a lot of other stuff that happens around these aws services that mox and even local stack are just not going to give you setting up the marks can also be an incredibly time consuming process [Music] the idea here is that if you actually want to mock properly you need to find every possible response that a service will give you to test it properly and that's a lot of potential responses and a lot of potential responses that you need to code in the actual content of so it can be incredibly time consuming added to that if you're not already doing unit testing as a part of this local environment you pretty much have to do unit tests to get this to work you need some way to execute the lambda code locally and using a tool like mocha or jest is a really great way to execute unit tests locally and the mocking tool ties into that because you can you can initialize the marking tool right up front with your tests send data to those uh to the lambda functions and have the mocks capture the requests and respond to them this is how the unit tests help you do that anything even if you don't use mock.js you're pretty much going to be doing unit tests at that point just to get your mocks to be up and running also it doesn't quite emulate the lambda environment accurately enough so it's still running on your local machine it doesn't actually quite emulate lambda so moving on to another approach which is very popular and we even use it at serverless and this is a hybrid approach and the whole idea here is that i might run my code locally but instead of trying to emulate or mock everything locally i just use the cloud services themselves so that the services are actually in the cloud i'm calling actual dynamodb actual s3 actual whatever else is that i need to and in this way i'm avoiding that issue with inconsistencies between emulation and mocks so instead of doing that i'm testing the actual services so it's a step in it's a good step in the right direction there's no need to emulate services which is fantastic you don't bog down your machine there's no complex maintenance that needs to be done and the code is still running locally so you can still use those local debugging tools that i mentioned in the beginning which is kind of an fte win um and at this point uh most people might be saying but isn't that good enough isn't this hybrid approach uh good enough for us to be able to develop our serverless applications uh obviously my response is not quite uh even with a hybrid approach there's still some kind of special configuration that you're gonna need to do to allow the code executing locally to point at those remote resources just running the lambda function locally doesn't it doesn't know where the aws account is with these aws services that it needs to connect to to use as a part of you know while you're building things and often this results in a development environment only configuration that ends up going into production as well you end up deploying stuff that's meant to be for development only into production which to me is a bit of a code smell uh some people might find this uh acceptable uh that that is a decision you need to make um i typically don't like the idea of taking stuff that i would use for my local development environment and putting into production even if i've coded it in a way that that stuff will should never be used in production i always worry about what if it is so yeah that's just a potential downfall of that and of course it does not emulate the lambda environment accurately enough so let me broach the elephant in the room i'm sure you've noticed me say this a few times it does not emulate the lambda environment accurately enough and this is ultimately one of the deciding factors for me when i'm looking at building my development environment i typically want my ability to develop and test to be as close to production as possible and this keeps coming up besides the fact that lambda the environment that lambda executes in versus the environment that your local code exists in well i potentially have different libraries installed there's been many times i've written lambda functions locally and using for example encryption tools and it runs on my local machine there's no problem with that as soon as i push it into lambda i suddenly have errors coming up with the missing file here there or wherever because it cannot find these libraries that i just assume will be there because it worked on my machine those can obviously be resolved you could just change out from one library to a different javascript module or whatever it is but ultimately you're still producing things locally that don't quite emulate what is actually running in lambda and that means at this point you are deploying into lambda getting the error you now need to make a change locally that you can't replicate unless you go delete libraries off your computer you have to now push this back into lambda and now you're doing remote development anyway just to solve this one problem the other side of this is that the way code executes in lambda is very different to how it does on your local machine even in a docker container lambda has a habit of shutting down as soon as possible it's part of why we love it it saves us money it means we don't have this thing processing 24 7 costing us five dollars minimum a month to sit around and do nothing lambda will execute your code and this is probably more prevalent in something like node in javascript than it is in the other runtimes but it does still occur in the other runtimes too um especially with the with the javascript call stack what often happens is that you have a request a promises generator the callback is generated that gets shoved onto the call stack in javascript and this is an example by the way you make a call to a database that gets stuck on the call stack your code continues to execute it ends it sends a return and in a local machine it means that that stuff in the call stack gets completed that request to a database gets completed sns gets called emails gets sent everything else that was queued now gets done in lambda however as soon as your function reaches the end and returns everything in that call stack is dropped um sometimes and occasionally i've seen a lambda a warm lambda that had stuff in the call stack previously be reinvoked and suddenly this stuff is executed again or before the actual code runs so you're in a situation where you have some stuff that you want to execute doesn't but sometimes does and you can never you never quite sure how and where this occurs um this has happened to me multiple times i've had calls go to services apis third-party integrations you name it and i run things locally and it's great i push it into lambda and it just doesn't work and then i'll spend the next hour and a half because there's no meaningful way to get feedback about why things are failing unless you know about this potential pitfall the problem is that there might be other potential pitfalls we don't know about but because we're stuck on our local machine we can't quite see that unless we're testing in lambda because the javascript callback anyway and testing serverless applications now thankfully there was a very nice uh talk about that so i'm not going to go into detail about actually doing testing serverless applications but i'm just going to go through a bit of a a a a summary of how this can be done so one thing to bear in mind is that okay i'm being chased okay let me get this foster okay so we have barrier the barriers are testing locally that we had in the past because of production being too expensive on our gone it was too expensive to give every developer a replica we can do that now because every developer can deploy to the cloud for free or very little if you've worked with silva's framework serverless deploy stage stage name or give every developer an aws account it's cheap it's easy it's actually free you can do with aws organizations okay similar is cloud native and this is something we need to bear in mind building our apps locally isn't necessarily practical anymore we are consuming the cloud and to make the best serverless application we can we should be trying to consume as much of the cloud services we have available to us as possible because we don't want to manage maintain and work with all this infrastructure and stuff that we have in the past we want to give that to our cloud provider and say you sort that stuff out i want to sort of my business logic and that's all i care about so the fact that we're cloud native means that we should probably be looking at developing more in the cloud and that's it am i okay thanks guys [Applause]

Channel: ServerlessDays

Views: 86

Keywords:

Id: m53xJILqLao

Length: 31min 2sec (1862 seconds)

Published: Sun Nov 07 2021