Anton Caceres - How To Build a Python Microservice Without Losing a Job

thank you dear friends it's so nice to see you let's get right to it did you notice that executing the same actions in different order can often lead to different results even if we take very very correct actions that normally get us to beer or some other wonderful results and do that in the wrong order we can get into a disaster how often have we heard our I should have done that first the famous thing I did a speacial applies in the software development because as we all know premature optimization and what is premature where is this point it is pretty hard to predict sometimes so what we're going to do today is we're going to talk about some not so obvious aspects of building the distributed system of micro services in Python and as well as in other languages without failing much because I know that's like yes if if you are full-time employed getting fired is pretty hard if you're freelancers gets easier fortunately we have now this whole micro service hype makes things much easier you can fail in so many ways so getting fired this is really getting there so I will be also sarcastic sometimes so if you recognize yourself it's not you I don't mean anyone I mean myself in the first place because I went through this whole thing and let's start simple we start with a stage zero before even the whole micro service thing starts imagine that the big boss is calling you hey we have to talk you think am I already fired what's happening no you're not he says it's time I was thinking I was reading report it's time that we go microservices at this point you might think hey but sir we just hear the Dressler saloon we accept booking something on the phone anyway like what are we going to split into micro services don't do that don't do that that's humiliating the boss he perhaps read so many Gartner reports about success stories he was thinking how he will brag in the bar with his boss friends I Hey Jimmy microservices now what do you have and you offer him some Django thing no let just do what he wants because otherwise if you don't then you failed all the stages zero you will not even get that job as a freelancer is particularly relevant and now you think I'm talking a bit extreme maybe but I have a real-life case I worked for one big and famous company which will I will obviously not name they hired me to fix their architecture and when I presented my first draft they said all right but we're a micro services so what's microservices is that a requirement they say of course I mean looking to the future now she's supposed to be an expert so good well should understand that micro services are not so much a tech solution as an HR solution when you have thousands of people and they have to cooperate on the same project efficiently that's a very good way to go or if the Lord is has such a difficult nature but you guys you have like one requests per second big time they were disappointed thought we are hiring an expert and you're offering us a monolith so let's just agree micro services take it as granted it's hype they will come don't resist it accept it and start forming you're happy and motivated team that is just easier to jump right into that right into the emigration how would you normally do such a big new beginning you would perhaps read a book and there are many good books you have a good selection also of the cloud container patterns available now published everywhere even for free but the book I mean man the book is boring who reads a book it's so 90s for God's sake we have Google Google is a new book Google is all we need why would you read 500 pages in advance your team will be so upset so if the team wants to just prototype you know freedom of choice and the freedom is a basic human right they will be upset if you don't let them prototyping you might lose your job so don't do that let them prototype because Google driven development is actually not that bad it's not as bad as Stack Overflow driven development who knows the difference difference is that Stack Overflow is when things are on fire so if it's Google they are sort of planning it in advance right now let's go a bit more serious now let's go deeper down to the stack stack you have to design it from scratch and here to be correct I have to say stack sip because we have a new fancy polyglot infrastructure we have many languages and it's not just about the languages because every language has on stack every language has on tools on build process and now we have many of this wonderful you know when you push Python code you perhaps do pilings you perhaps run tests you package it somehow flake it's whatever build a docker image that's one way of doing it now imagine the next thing you have is Java you'll get to know me even the beauty of maven or ant you will pack things differently some jars so worse or whatever they're using and now imagine we have front-end as well on the stack we have JavaScript wow you would have to minimize it add shims can catalyze stuff Babel other things a wonderful life and then if it's really really front-end and the webpack goop grunts oh that is so much fun I have a perfect illustration for this I took from the GS Congress in Munich it says there are milk products that last longer than some JavaScript frameworks that totally reflects the situation of the front-end these days so yeah you're psycho look perhaps like that and just share it with the ops ops will be so happy to see it they either jump out of the window or throw you out of the video so get prepared get your team with you so that it won't be you but of course of course you can always say it's better than the monolith and old ugly monolith this nasty Jenga tower standing out there ready to fail we're going to fix this with microservices and it will be made more elegant there is a surprise waiting for you because if you if you migrate from here to micro-services 90% of cases you'll get many little junket hours instead of one thing along with its own stack on with its own box and with own hacks to keep it somehow stable and alive you know now imagine that there are just we have micro teams right imagine there are just one or two people working on each Jenga tower it's a good thing cannot get fired because they depend on you heavily right now so they will probably not just not fire you so the smart boss would probably vaccinate you against everything in this world because like a better business depends on every every little piece of this and you're basically and replaceable I love it as a result what I call it is an epic zoo of Technology imagine the front end is and react statistic is an R and then you have some admin side HP legacy you have accounting made in Java you have to maintain that whole thing somehow and this components they they bark some components meow and another components role and sorry I forgot the fish fish is important but it just writes logs so can keep silent now imagine how can you possibly manage this whole thing how can you keep this component sane and well for keeping it saying we are going to the next chapter even deeper even more concrete we are going to repeat some wisdom of a bigger companies that did it in the past already imagine the new team member comes and says I don't care it's all in Python I want my new fancy micro service in JavaScript what you have to do to offend him or her should be polite and you just use the wisdom of Netflix Netflix had this issue they had the whole thing done in Java it worked perfectly until they hit certain architectural limits so they had to split into micro services which were Java micro services and then they grew grew grew and eventually they had to deal with other languages as well so they developed a very very smart strategy that works for them at the beginning they said of course at Netflix you're very welcome to pick any language you like you have a total freedom in doing that but just just in case just in case that language you'll pick will be Java you'll get the failover you'll get the discovery you'll get logs you'll get monitoring all for free automatically but if it's not Java well you're very welcome to implement it in your language of choice and at this point most of the developer said maybe Java is not that bad option what I wanted to say is that at the first stage all right let's say you have to split your monolith into many components but that doesn't mean you need to have that zoo right away you can still stick with one or maybe two platforms that will be well tested that will have a well-defined process on the up side of building things and testing things and on deploying things so it costs a lot especially on the in the beginning to integrate every new process in your architecture so if you can avoid it just avoid it at the beginning and you can pick just one reliable well known language such as Java and work with that kidding kidding Knorr so let's say we take Python we need still to decide on the exact tools that we are using in Python to write micro services and here I'm just going to quickly jump over the most popular frameworks and choices that we have let's start with Django that's not a joke if you're damn good in Django you can write a microservice in Django I know people who did that it feels to me like going grocery shopping on a truck nonetheless nonetheless yes you can do that steel because Django is still like this very very flexible plug-in architecture which you can customize we can throw use the things out and you have wonderful pieces like an admin panel they're automatically but the problem is that if you take Django by default it has so many components so if you think of that actually each component could be a separate micro service so it doesn't really make much sense to use Django as it's shipped to you it only makes sense if you cut things you don't need out then it's fine and speed wise I mean we don't have to discuss Django we all know its advantages and disadvantages but it is still doable so here many would suggest flask flask everyone loves last basket so minimalistic it's fit so well the whole idea the whole concept of micro-services because it does just one little thing and it does it right I like flasks a flask is is okay it's not the most fast web framework but it's a conference what I like in flask is a synchronous networking and this is particularly useful microservices because when you have a chain of requests from one micro service to another and they are blocking and one thing fails and the whole chain is blocked and does not serve more requests because potentially I know it gets out of the workers in the pool that's bad that's not cool so I strongly prefer and suggest you to use the synchronous tools so that you are not blocked if one of the components is blocked and you don't have a limited amount of slots of the parallel requests that you can process at one point of time so there is a solution there is squirt quarte is the replica flask is just a synchronous copy of last let's say I didn't check it out in production I just know it exists I played with it it has a very same API as flask does I think they are developing it further now but it's a good option nonetheless as I said I didn't check it in production flask I did so if you were a flask person and I know flask is a much better framework because it has a cooler logon and its minimalistic it's like 500 minimum thing does all you need looks fancy for those who do not need to compensate you know with a big scale totally love flask now I'm just going to quickly talk about other options there there is hug who knows hug hug is great hug is based on Falken Falken they say by free by the benchmarks that Falcon is the fastest web framework in Python by amounts of requests per second you can handle Falcon wins it's synchronous but it's very fast and then hug is a sort of abstraction over that simplifies building API it's like Django rest for Falcon makes developing restful endpoints very easy generates you the common line interface right away because remember your micro service is not necessarily they have to communicate over HTTP it could be that it's very useful if you have something like Amazon CLI available right away create a normal restful endpoints you have a console interface automatically that's pretty handy I used hogging production does a great job then there is tornado tornado is fifteen year old now as far as I remember it's super stable it's synchronous it's well tested well-documented it's like more more the older version of async IO let's say that supports Python 2 as well which for some cases because usually when you come to a project that wants an architecture upgrade the chances are that they have a lot of components written in Python 2 so with tornado you can have a fancy synchronous communication right away since 2.6 they sort of a simulate the yield from or async/await thing with just normal Python generators so you can use yield you can use yield with Python 2 and it just works very similar to the way I think i/o does I'm not putting a single on slide because that is so obvious like that's the default thing you'd go with a HTTP and they sing heyo that's now the default way to go now the project grows and thanks to Python of course at some stage we hit the points as Netflix did where we cannot resist getting new languages in your stacks anymore so what can we do then then we should use the container patterns and there are books published their articles published and also many many videos on YouTube about that I'll describe three most popular patterns that are used and it I checked personally let's start with a sidecar sidecar is sort of an extra container that he puts next to your main container that would enhance it to be more precise here is how it looks like imagine that you have a main container which is the webserver and your sidecar container is the lock saving application why would you need to separate it you might ask you can have your main application and something like a century agent that collects your logs or Cubana or whatever the trick is that every web server even super old one and super legacy one can write logs too hard drive and not every old framework has Center integration so we go a bit another way and we say let's just assume every app can write logs to the hard drive and we take another container that would pick logs from the hard drive and send it to whatever cloud aggregation system this way we separate totally the responsibilities of these two we say that the web server should just do the web related job and save logs as it can and it's a job of the sidecar container to pick this lock ups process do the whole routine find the right cluster to submit it this sort of work this is the first pattern that you can use and the second one is ambassador it's a very similar concept sort of a big important fella that you have to speak to instead of speaking to some even bigger entity this is especially useful when you are dealing with some remote resources like a database or a Redis cluster ambassador will hide it for you ambassador resides on your localhost it runs next to your main container and you only communicate with the Ambassador it simplifies a lot because in your super huge application you perhaps have a super huge Redis cluster that every component that speaks to has to maintain you either need the IP or DNS and if it's failing then you need a new root so this is all job of the Ambassador and the master is always on localhost the same port so if anything fails it's a job of an ambassador to get the new root find maybe do some graceful degradation in this case or caching things as well can be done on the Ambassador level so this is also a pretty common pattern very useful when you're writing big big distributed systems it is particularly useful because as I already mentioned when the new language comes in it does not affect the main container you can have a main container in Java and you can have your Ambassador in any other language they communicate over standard protocol standard ports they are totally running independently now finally the adapter the adapter looks like like an ambassador but the reverse ambassador if the ambassador pattern represents an application your main application with a simplified view of the outside world outside cluster then the adapter does the reverse it simplifies the view for the outside world of your application to give you an example imagine the monitoring case like perhaps we want to have a health check on our main application and different frameworks different applications serve from different protocols they have different ways to check if the app is a life or it's down some would provide you a URL like slash health so whatever some would write in a file in UNIX that would say file is there then I'm up and some would just not do anything you would have to do PS outs and check if the process is running in the UNIX machine so we can use adapter to help the outside centralized monitoring system to get the status of every component because we will ship this monitoring adapter next to our main application and the adapter will know how to check if the main application is dead or alive it would know if it should check a file or check a URL and so then the external the external part the monitoring system would just use the same way to ask the adapter if the system is running or not this way you can also scale framework independent and you can keep the same monitoring system on a very very very diverse zoo good now quickly going to the functionality after we know the patterns as we already discussed micro service usually has some business logic like the main thing that it usually does either serving some crude operations on the database or doing some monitoring or something else and there are always communication functions which are repetitive across many different projects it makes sense now to take this communication functions and put up in a sidecar that I already described to you because they are very repetitive they do this failover and discovery and graceful degradation thing it looks much more elegant like that but we still have one problem now we uploaded all of the functionality of network functions to the sidecar however nonetheless how should different application find each other you packed it into sidecar but what exactly is the sidecar doing how is the network communication and discovery happening you need to communicate not only with the database your components perhaps also need to communicate with each other and how exactly can this be done elegantly you can of course would ambassador for each service but you know it will quickly get insane you can you can break things and lose your job so you need a better way of establishing communications not just with a big Redis cluster but also with the components talking to each other like small dots and here we can finally introduce the concept of a service mash it's a new step it's yeah it's a bit centralized it's a centralized component that keeps keeps track of which service is doing what's and available where and so the idea is it's this that the communication between our little our little services do not communicate only over service mesh they just use service mesh to find where are other components located they still communicate directly so this is like DNS on steroids basically you can say I can just use DNS at the beginning but DNS depends on caching and in such an architecture you cannot afford caching too much because if a system fails you need a very very quick replacement for a failed system and that's exactly what service mesh does so you can think of it just as a better DNS system and to be more precise what exactly is doing I have a list here like you normally offloads tasks like routing access control it's another big thing that I'm not discussing right now but when you communicate with in micro services you need to decide on the common way how if you encrypt the data how exactly how do you decide if one service is allowed to ask another service of a particular operation then there is always a context of are you asking one micro service to another on behalf of user or you just need some meta information all of these little things you have to do repetitively in every new micro service you bringing into your architecture and not to do that you upload this repetitive tasks to the sidecar and the service mesh is just maintaining the list something like the access control list routing list task list and so on good finally I want also to say a few words about the failover people say graceful degradation and short circuiting there are many fancy ways of saying how would you elegantly tell your users I'm down it's way more easy to bring the system down when you have many little services because it's a chain if one element of a chain fails I mean if instead of showing error 500 you are showing a message I'm sorry we're currently not working it's more elegant but it's not good so instead of instead of doing putting effort in the short circuiting and failover I would strongly suggest you to put your effort in better logging and better tracing there is an open try standard that was developed especially for this case when you have a big system and it's it's hard to it's hard to debug it it's hard to trace where exactly did the request fail instead of putting your effort into excuses I'm sorry we couldn't do it trace it log it make a have a system even when we have here sponsors what was the data dog people like they do it I use data dog for that I'm not advertising them but also New Relic does it invest your time they are built a reliable system where you can easily trace every requests and every time you can say if something fails it was fault of this component because without this you will produce certainly eventually lose your job thank you for attention and I wish you a very nice zoo of your microservices [Applause] Thank You intern for this refreshing talk and we now have time for questions so we have microphones on the side or I can also move okay so in the meantime while they get inspired I wanted to go back to your recommendation on basically the frameworks and there has been quite some articles about why and talks about why frameworks can also constrain you a lot especially when you want to iterate over your products or let's say add new features because they basically fix the way business logic is defined so would you have also some recommendations where on maybe just smaller libraries or how you how you would approach but if basically writing is not an option yes of course well one friends is things like lambda functions on AWS many people do it many people do it successfully there is basically no framework that's just a function as long as it's simple it works however if you are not so much afraid of leaving that project and you think that there should be another person replacing you eventually frameworks are like like a common patterns that other developers knows if I leave a project and they tell you it's just the Django rest endpoints and you can find right away it's supposed handler it's a get handler and you can just continue working right away if you do a customized system obviously it's harder for new people to get into it on another hand there is no work you love using a framework especially in micro-services because even tornado it's quite minimalistic nonetheless it has also as Jenga does many batteries included like CSRF protection and so on which you normally would not need in micro-services so your rights it's always a trade-off when you pick a framework you make things simpler for yourself and also for people who will be working on the project after you if you do it yourself you usually can tailor it better you maybe can do it even more efficient but there will be difficulty in maintaining it when it grows and then don't forget the testing you would perhaps test things that be differently when you writes micro-services because it's not so much about unit tests in microservices as about integration and acceptance tests you need to check the whole chain not just one thing and hear testing frameworks that act predictably is sometimes just simpler than testing custom code that is not so predictable for others not for you for others that test their components speaking to your component so yeah but there is no black and white answer thank you thank you the presentation as for father said discussed in his blog post about micro services the reason s for this error propagation a synchronous communication and the problem is that when you add micro services to your system probability of going down actually increases because we with with each micro service that can go down so the solution is a synchronous communication between micro services it'ld and it may be HTTP or anything else but did you try to discuss it what can you tell about about switching to a synchronous communication if you mean just switching from synchronous HTTP to as synchronous HTTP it will not change much the protocol is the same it's just what I meant at least in my part was that the framework that the webserver behaves differently instead of blocking one worker doing one request you have the i/o loop that would handle that requests as soon as it's not hitting any i/o boundaries and if you hit on your boundaries which you always do in micro services because you're always waiting for either a database or cache or another micro service it can serve other requests as well that is what I meant with synchronous communication I do not see the effect of this on the error tracing because the protocol is the same it's still HTTP on as a client you should in the best case not even notice it is asynchronous so it's synchronous regarding the error tracing yeah and again I didn't mean error tracing on the client level like HTTP error codes like error 500 or whatever else as I meant error tracing inside the application because you your request is passing through so many systems it's hard to to match by time for instance that's a typical first step that developers do when something fails they try to find logs on that point of time but it is going so fast that you perhaps will have hard time figuring out because at exactly the same time in many many services many many things could have happened at the same time stamp so it's harder to do this chaotic time based debugging and you need more centralized tool that collects all logs and that somehow is marking every request with some headers which which are then passed around from one micro service to another so that you can trace it from the entry point to the exit point what exactly happened where that is what I meant was error handling sure that's correct but I was asking more about dealing with unavailable services for example what do you do when you issue an a request to a micro service that should be there but it's not there it may be it's not there for like 10 seconds and the question is will your system be failing for this 10 seconds or will your system be silent and be able to recover of course we will have to stop here now so I would suggest you to continue that during the small break we have the rest fix and then once more please and then we have a great

Channel: EuroPython Conference

Views: 7,150

Keywords:

Id: uYgla2-mxMc

Length: 30min 16sec (1816 seconds)

Published: Mon Sep 23 2019