GopherCon 2021: Arish Labroo - How we Go at Zillow

hello everyone i'm arish labru and today i'll be talking about how we build go services at zillow most of my slides are code snippets actually which you can uh go at this repository if you have any feedback uh zula has a lot of services actually a lot of services built in different languages and different stacks and a few years ago when one group started advocating using go for production applications and services and you know it grew in popularity more and more teams started using it and we started seeing some patterns and practices emerge which we wanted to standardize on zillow has an amazing developer experience team which you know in addition to managing our deployments containerization ci cd observability tools and stuff they also manage archetypes for all our uh stacks so we started out uh with building our archetype for golang with certain goals in mind right consistency was huge you know every service every deployment should look and feel the same you know so that if you're jumping across services uh there are no surprises we like a lot of services you know uh micro services some uh more broad scope services but a lot of services you know they should be maintainable the underlying components a lot of cross-cutting concerns change more often than we would like them to but you know anytime something has to change or you know we need to add a more core feature it should be easy to distribute that across all the deployments you know if you have hundreds and hundreds of services uh small changes like uh you know adding new traces or new metrics uh become a year-long effort so you know we wanted everybody to get that automatically uh resiliency is uh is a must for a production grade zillow application uh reducing the blast radius uh you know minimizing the impact of errors and stuff like that observability you know we need to have ability to comprehensively observe our our production applications logging and metrics are obvious uh distributed tracing you know you need to have an end-to-end comprehensive visibility on how your application is performing you know throughout the stock you will see how pervasive tracing can get and you know you have to be very conscious about opting in at every step uh testability uh you know every application has to be thoroughly tested uh unit testing support in go is already great you know what we found out your bigger value was uh observed when you uh wrote end-to-end integration tests right uh so you can you know if you can uh don't worry about all your fringe dependencies and then test your application and business logic comprehensively uh in your test uh that provides a lot more uh confidence uh so with these things in mind you know we set up on an architecture most of our services most of our deployments look more or less like this we have an explicit transport layer uh you know you have your service logic the application domain and then you have all your dependencies you know dependencies are more or less the same across all applications you either read or write from database you have to emit events or you make http call to uh downstream dependencies uh transport layer is interesting because all of our services out of box are 100 transport agnostic what that means is no matter if you're calling your service as a you know http rest endpoint or an rpc endpoint or you're passing messages over a durable transport your service should not care you know all the transport specific logic terminates in the transport layer unmarshalling json request body you know decoding binary encoded protobuf messages reading from kafka or sqs should all be transparent to your implementation you know all the expectations that you can have from one transport you should have from the other transports too you know we actually have services in production which are used both uh with rest and rpc and they all have workers you know which process the same requests over a durable transport we use http primarily for reads and most of the rights are through sqs or kafka uh you know separating your reads and writes cqrs style and even if you make a write request over http we immediately offload that to uh a queue and process it asynchronously and it gives you nice reliability guarantees uh so in this talk we'll be primarily focusing on you know how we standardize on these components so that individual developers hundreds and hundreds of developers can focus on the requirement on the the uh business value that they're providing while a lot of these cross-cutting concerns just light up for you uh let's let's start with the server uh you know http server rpc rest however there's an http server and setting up a server in go cannot get simpler you know you have the http default serve mux you have http server you just listen and serve and you have an endpoint which is handling your requests this is not production ready uh you know there are a lot of things that are a lot of non-negotiable requirements that every server has to implement you know you need to have timeouts on your uh connection request handling has to be uh you know capped with certain time so that slow clients or slow requests don't bog down your availability you know you have to handle graceful shutdown os signals you know giving some buffer for in-flight requests to complete and not serve any more requests you know simple you just listen to the signals on a separate routine uh distributed tracing again distributed tracing is very persistent like you know every hop across every transport ingress or egress you have to be cognizant of uh propagating your traces propagating your context you know because if you if you miss even a single hop uh the traces are rendered useless right you know they will get new trace id then there's no correlation and you don't have uh end-to-end visibility visibility into how a request performed so uh you know you can wrap your request in a open tracing net http middleware where every request automatically creates is processed under a span you know the tracing context is propagated in the request context or like we you know we use a dog for uh tracing uh and visualization uh amongst other things uh they provide a serve mux that we use which is similar in behavior you know it wraps your request creates a span adds a few additional tags which just light up in the trace visualizer no matter what tracing tool you are using you have to be aware that every request needs to work under a tracing context health checks and probes our infrastructure mandates that we have probes available uh on every server every live server in production uh health check live nearness check readiness checks and stuff and in addition to just returning uh you know like an http okay response at times you would want to check that your dependencies are behaving you know your database is available your downstream dependencies are available so you know these health checks these probes need to be extendable uh specific to application requirements uh swagger right uh we we describe most of our services in protobuf uh and we use twitch uh twirp rpc framework the beauty about twerp is it does all the content negotiation for you automatically and you know you can serve uh rest as well as rpc endpoints from the same server it takes care of inspecting the uh headers and on marshalling or decoding the request so that you don't have to worry about which uh what was the message format uh and since our services are described using protobuf we use the protoc swagger uh gen plug-in to generate spagger specification and use uh an open source http swagger package to host the swagger ui so every every http server has a slash swagger endpoint automatically readily available without individual developers having to worry about this you know great for documentation ad hoc testing you can you can just simply call this server and there's a bunch more you know every request we would like to be profiled automatically logged metrics uh you know you need to have a metrics end point where the prometheus server can scrape uh with some uh metrics already registered uh collectors already registered automatically uh you need custom request context propagation so if you have uh like some configuration flags or user id or user token available in the request headers it would be nice if they automatically light up in the request context authentication security headers and stuff so when you take all of this into account a simple server becomes a few hundred lines of code you know and this is all required uh features not recommended and not nice to have must have features that every server has to have and we don't want hundreds and hundreds of developers have to duplicate this every single time again nothing special about this everybody knows this we just don't want to lose it you know in any service at any any hub so what do we do to standardize server behaviors we create a simple server abstraction it's simple abstraction around http server it has a handler you know which is your serve marks you know we use data dogs tracing wrap mux if you use gin and you can adapt your jin engine to this interface or gorillaz mux it just needs a simple handle func function uh we pass in a configuration everything about the server is configurable you know where is the swagger file hosted what are the timeouts and what port and all that stuff uh you know configuration can be read from environment variables or configuration files and whatever and the server has a simple serve this is we spin up the this is where we spin up the http server uh correctly configure it add all the default routes uh listen and serve and also handle graceful shutdowns right so now uh ins and server has a newer a simple constructor uh which will uh give you a default server instance and you have uh functional optional parameters that you can pass to modify behavior and then using this new server this incomplete implementation of an htc server server becomes this you know your new upper server and you add your application specific routes this server is already uh more featureful more resilient and more complete but this is still not production ready you know you must have noticed that the server has other members like the logger the tracer configuration needs to come from different sources metrics handler and stuff like that we don't want every individual application to worry about composing these dependencies too and we don't want the server to be opinionated about how it logs or how it traces right the logs need to be contextual uh they need to be correctly scoped to your request you know tracing needs to be configured for your application for your environment you know configuration sources can be different so if we had to uh request either make these as a required parameters in the new server the constructor or we make sure that everybody passes this as uh options that's still a lot of overhead in every every service uh so instead what do we do you know how do we uh simplify and standardize on creation of the server we have a server factory you know if you have an immediate aversion to factories and go i promise you for standardizing and consistently creating opinionated instances of certain common types factories in a great creational pattern actually factory has a simple create which gives you a server factory has a lot of these members the loggers the tracers and all that stuff which somebody needs to compose you know like for example the factories constructor would look something like this where all these parameters are required how a factory is actually composed we'll talk about that in towards the end how we do composition in our services but let's just assume there's a factory from which you can create a server while still keeping it configurable to your application's requirement and now instead of initializing the server yourself you have a factory you get an instance of a factory you call create and handle your application specific routes now this is 100 production ready feature full http server you know it has timeouts shutdowns context propagation tracing all the probes swagger and all that good stuff for you automatically available and the best part is if we wanted to introduce uh you know across uh across the board features into the server you know exposing a new endpoint that by default should be available everywhere to or you know we need to enable ppro for example we don't do that in production but you see the point right there is one place one library where we have an opinionated server which we can change everybody just updates their package and everybody gets this feature for free the factory is still customizable you can pass an option for example here we are extending the health checks to uh ping the database and see that the database is available but more or less this is how a server initialization works at google at zulu uh so along with the server another deployment model that is very popular at zillow is a worker worker is a long-running process which is reading messages from kafka or sqs or polling databases at a regular interval uh you know cron-like operations uh we build a lot of systems uh as event based at zillow so we we pass a lot of messages over over durable transports uh and the best part about workers is you know you can scale them horizontally reading from different topics or different queues or you know where different request types can go over uh different uh queues most of our services are like one server you have one one http server which you can also scale but we have a lot many different workers so uh setting up these workers it's uh it's you know paramount that it is very simple and seamless uh to be able to spin up these workers easily let's take a look at kafka worker for example right for a kafka worker let's consider that we have this kafka abstraction which takes care a lot about all the intricacies and the complexities of reading and writing from kafka is handled by this kafka client it has a very simple exported surface uh you have a client from which you can get a reader and from the reader you can read messages one at a time right so let's say this just automatically lights up from somewhere the composition of this package follows the same pattern that we'll see eventually but for now let's just say there is a kafka client if you want to build a worker a simple implementation would be an incorrect implementation would be start a forever loop read messages one at a time and process them uh this is not comprehensive this is not production ready this is not deployable at least at zillow uh you know it doesn't do timeouts uh similar to http requests where you don't want a slow request to bog down your server we don't want slow message processing to to create a bottleneck in your worker uh you know simple implementation uh you have a with timeout which is configurable context with timeout which is configurable you need to handle os signals too uh circuit breaker is is something that we ideally want every working implementation to have so that you know if message processing is failing above a certain threshold we want to slow down or maybe even sometimes stop processing more messages you know to give some breathing room to the dependencies if they are having issues not overwhelm them or you know if your worker itself is having issues you know having more messages go into dead letter queue or having to move the offset around in kafka is irritating so ideally we have some circuit breaker implementation so we reduce the you know the blast radius we use sony's go breaker library which is completely configurable very simple to reason about we again distributed tracing you know distributed tracing will show up literally everywhere uh just like http request where we had a middleware uh when we are processing a message we need to extract the tracing context automatically and you know light up and do the processing underneath the span we use open tracing which has a text map reader and a text map writer interface the implementation is transport specific you know for example kafka has the tracing context in the message headers sqs has its attributes you know sometimes you have it in the request body itself so extracting that context creating a new span under this context as a child of the parent context parent span id and then passing it to the actual processing context so that it can propagate it down even further is very important and you know it's not optional you know we need it to be done consistently every time so we uh you know we extract the context for you and when we call process the context already has the correct uh tracing context in it uh concurrency uh you know concurrency is best left to your application right that's that's the uh recommendation but uh doing uh concurrency consistently has a lot of boilerplate uh even though go makes it super simple you know uh but you know having a concurrent evaluation of circuit breaker features uh error groups uh context uh propagation uh having bounded number of go routines that are running at the same time uh you know if you're using sqs for example you want at least the max number of messages being processed concurrently otherwise the revisibility timeout kicks in for kafka you want at least the number of partitions that you have to be processed concurrently so you know if we leave this on uh at each hundreds and hundreds of deployed services to do it consistently there's a lot of boilerplate that everybody have to uh implement you know chances for errors go up uh so you know we just do uh simple concurrency constructs for you behind the scene and a lot more logging profiling metrics server you know you need an http server exposing the metrics endpoint for prometheus to gather metrics custom context propagation like we talked about in http uh user propagation you know if you have a user in the header uh or sql attributes you would like it to just light up as i said early on the implementation of the service is completely agnostic to the transport so whatever expectation the service implementation has we need to uh keep it consistent across transport so all the behaviors that you would see uh for a worker uh is exactly you know you can make the exact same request over a worker or over http and the behavior and there will be no change in your service implementation uh error handling for example is very important worker right you don't want a panic in one messages processing to bring down your whole worker so when you take all of this into account this worker becomes again a few hundred lines of code which we don't want everybody to have to duplicate hundreds and hundreds of times just like server to solve this problem we create a worker abstraction uh you know simple uh does a lot of heavy lifting for you behind the scenes you just have to focus on your application specific logic uh worker has a simple factory too you know which takes care of initializing consistently all your dependencies the factory has a simple create which will give you an opinionated correctly instantiated worker instance and worker has a run server has a serve worker has a run you just pass it a processor which is nothing but a function which takes a context the correctly scoped context and uh a function basically the message you know the message that you are processing for this request and it takes care of timeouts concurrency tracing error handling uh all that good stuff and usage is also super simple instead of doing this in complete implementation you create a worker from a factory and you just run it pass it to your application specific processing and you are good to go this is 100 production ready worker uh all right so you know server and worker are two more important deployment models that we have at zillow we we actually you know we have a lot more workers uh along with all our servers but all of these services have common dependencies uh let's talk about databases first you know quickly how we do uh databases uh the dv sql package makes it super simple you know open a connection and do your database operation this is not production ready uh you know you're ideally supposed to reuse all your connections one common pattern that we see is hey just create a global variable for your connection and then this it will be application scoped the standard library will take care of uh you know the lifetime and retries and all that stuff or creating new connections all that stuff but a global variable almost uh makes testing end-to-end integration testing impossible right uh tracing you know if anything in your application you want to trace comprehensively if you want visibility into is your database operations right how is your database performing uh we want it to just light up for you there's a lot of configurations on the connection that you would like to set up based on your applications requirement you know we want you to be cognizant about it you know uh we have some sensible defaults we use sqlx uh you know just because it's easier to work with strongly typed structs uh and we want to use the contextual members right the db has select and select context for example we would like everybody to use the select context always so how do we do this we have another abstraction we have a provider a very simple implementation which gives you a database sql db instance provider cache is the instance for you you know the key being connection string a few other parameters simple mutex around this cache we use data datadog's db tracing library to to wrap database operations uh with the tracing uh context uh and you can set up all your configurations here uh consistently and uh you know you just every time you need to make a database operation you call this provider to give your database instance what this does not do though is the contextual members right we there is no easy way to mandate that you only use uh the contextual member out of the package you know one way is to create your own abstraction which only has the star context members but we didn't want to create our own abstraction around database so we actually use linting a tool that our teammate wrote which uses the analysis package totally configurable rule based and one of the rules is that you have to use the context members and you know it runs as part of our ci cd pipeline and if you're not using the contextual member it will prompt you all right and usage is very simple you ask for a db instance from the provider and then you use it and you get you know timeouts configuration tracing uh all that good stuff for free all right every service has to make an http request like almost all the services have to make an http request usually reads you know as i mentioned we use http primarily for reads but even if you have to make a write request through http could not get simpler you know use the default http client make a request uh this works uh this is not production ready uh you know you would like to be uh you'd like to be aware of you know timeouts that you set up on this request you know uh by default there's no timeout as i mentioned we use http primarily for read retries for transient errors is a safe default assumption and we don't want everybody to have to implement retries it's simple but it is duplicative it's boring and you know you have to worry about jitters and back offs and all that stuff so uh you know a good comprehensive implementation of retries for which is freely available to you just via configuration uh is ideal uh tracing you know you have to make an http request you have to propagate your traces right super important and other custom context propagation so what do we do just like everything else we have an provider which will give you an http client we use hashicorp's retriable library to give you a retriable client where all the behavior is completely configurable uh as i said you know don't uh we don't retry posts or writes you know because then you have to worry about item potency and all the other application guarantees but transient error for reads is a safe assumption but again this is configurable we set up the configurations on the client like timeouts and all that stuff what this still does not do though is anything about trace propagation right it does not participate in distributed tracing the challenge there is there is no good way it is that we have found out where you can intercept every outgoing request automatically and inject uh or add request headers right there's a round tripper interface but uh you don't you're not supposed to modify the request in it uh so we provide a simple helper you know where you can pass in an http request and your contacts and it will uh inject headers into the request for you uh also as i mentioned we use uh twerp the rpc client the twerp generated rpc client actually does not require an http client type instance it requires anything which has a do method right so we create a simple wrapper around http client where the http client inside the wrapper is the you know the retriable uh completely configured http client which we use uh and we provide the do method which basically creates a span for your request injects the tracing header using the open tracing uh http request reader carrier and now every request has retries timeouts tracing uh and all the configuration current configuration correctly set up uh and you know you have a way to get a wrapped client from the provider usage is super simple instead of using the http default client you get the wrapped client from the provider pass it to your rpc generated client and you have all the goodness for free again where does this provider come from right we talked about provider kafka client worker factory server factory where do all these things come from we'll talk about that in a little bit we write to kafka a lot similarly you know every message that we write to kafka has to have some consistent shape and structure we want some certain headers automatically added like a unique identifier timestamp the tracing context needs to be propagated over kafka so we simple abstraction you know we already talked about the kafka client it has a writer when you write pass it the context and it does a lot of goodness for you the actual implementation of kafka right you know we won't get into it there's a lot that goes in there too uh but let's just assume there's a kafka client which does a lot of the heavy lifting to interface with kafka but this is a simple do this is a simple write method which does a lot of consistent message passing for you again using open tracing's text map writer interface all right so you know we talked about all these components which at ingress and at egress a lot of we talked a little bit about you know context propagation is important uh tracing is important uh metrics you know we want metrics to be easily uh emitted statsd or promises whatever you're using it should be configured and set up for you automatically logging is ubiquitous ubiquitous and we do logging consistently you know every logs need to be contextual every logs need to be consistently scoped right so we have the simple logger interface that we use throughout all our services uh the only point to note here is the they take in a context as the first member so when you actually emit a log it has a lot of fields already automatically added with your log messages we use uber's zap library for logging you know super amazing super low overhead super fast super configurable you know you can shape your log messages however you want uh you can add initial application scope fields at the initialization of the logger you know the application name environment process id host name and all that stuff in addition to whatever fields you know the infrastructure is going to add and then you know you create a new instance of this logger which holds on to the zap logger as well as a tracer so that you can inject trace information with all your log messages uh you know like if your logs can be easily correlated with your traces it's super powerful uh so we would ideally want the span id trace id and all that stuff to be emitted with the uh each log messages automatically and you know the good thing is that you know you can go to your log visualizer and say hey give me all the logs for this request id and you get an end to end comprehensive view of how everything is behaving uh super powerful right so you inject uh using open tracing uh tracer which has been correctly scoped to your application and when you log a message uh in addition to the message that you're logging you get you know like 30 odd fields also added into your log uh configuration you know every we use a lot of configuration everything at zillow every service has uh behavior is configurable we like to have strongly typed and localized configurations right so many many many different small small configuration structs which can be hydrated from many many different sources for example host here can come from environment specific configurable configuration file uh api key comes from uh you know uh environment variable or hashicorp vault or uh aws sequence manager wherever timeouts can be uh you know some they have default value or they come from another environment variable so you know we have multiple sources for uh configuration and believe it or not the configuration sources change you know we have go we've gone through like console world secrets manager parameter store multiple times so we don't want everybody to have to worry about parsing these configurations they should just light up for you so you just create a configuration struct uh and you give a newer for this truck which takes this app configuration dependency we'll talk about app configuration in a second and you say hey app configuration just load this configuration for me you know go figure out where all these members are going to come from and give me a correctly initialized configuration instance and appconfig is a simple data structure it's a map which stores the raw merged configuration for that particular key in the newer for the app configuration we basically use the conflate library merge all the application configuration sources and just hold on to it in the map and when somebody requests a particular type we use reflection to get the configuration name and unmarshal the raw message that we were holding on to now this is you know this all this unmarshaling gathering the sources reflection is just once uh per application so there is no per request cost here and you get strongly typed configuration which is super helpful when you are doing testing you know because it's just a struct you can pass in whatever struct and modify the behavior in your tests rather than having to set up environment variables in your test which gets irritating all right so we talked about you know how we have all these uh crosscutting components available to you for you to use which offer a lot of consistency resiliency tracing and all that good stuff and we don't want everybody to have to worry about the implementation details about a lot of these functionalities and i mentioned we'll talk about how these are actually composed in the end how are they actually composed right you know because to initialize them correctly with all their dependencies is a lot of work uh what do we do let's actually take an example right uh it should help showcase this much better let's say we have a simple service you know which on every request has to do three things make an http request uh write to a database and write emit an event to kafka right not completely hypothetical a lot of our applications actually do things along the same lines if you had to compose this application yourself you know this is what you would have to do right initialize your configuration then your logger your tracer your kafka client your server factory your db provider and all that stuff uh and handle your request with all these initialized members this is correct this is 100 production ready but this is almost impossible to maintain you know for example imagine the redis client introduced a new dependency a breaking change because it always wants to consistently capture cash hits and misses right and you change the newer for the redis client now this is a breaking change for every application all the hundreds and hundreds of services have to make code change to satisfy this new requirement and if you make it optional then nobody will pass it right and then you have to go back every team hey can you please do this too and it becomes a multi-year effort so we don't want to have to have that overhead across the hundreds and hundreds of services that we have instead we use wire according to the documentation wire is a code generation tool that automates connecting components using dependency injection the beauty about wire is in its simplicity like you know if you have used dependency injection frameworks in other languages and stacks they have a lot of uh bells and whistles a lot of features and runtime inspection and stuff uh which almost always you know end up biting you why does none of that stuff wire is simple 100 human readable code generation i'll give you a small example let's say you have this dependency tree where foo depends on bar bar depends on boo and boo has a simple newer right if you had to write this code by hand you would initialize a new boo you would initialize the new bar pass it the boo you initialize the new foo pass at the bar and now you return a foo right simple but imagine if this dependency tree was like n deep and m wide it would get very irritating very fast and almost impossible to maintain instead uh we create a wire initializer right uh we create a set which has all the providers providers are nothing but a handle to your constructors your newer functions which says hey if you want to create a foo use new foo if you want to create a bar use new bar if you want to create a boo use new boo whatever are the parameters that are required in this newers wire will take care of initializing them for you and passing them along and then you create an initializer which will say which will tell where hey create a vu using this set that i just gave you right and it will know how to write this code how to write emit this code this is actually generated code uh you know this is exactly same what you would have written by hand but this is auto generated now imagine if this dependency tree was huge it's a big benefit because you know if there is an underlying change to one of the core packages you just update the version in go mod run wire again and it takes care of it you know you can actually put this in your ci cd pipeline to make sure that you are using the latest and greatest core components and every service every deployment is consistent always so how do we use wire you know we provide a zillow wide common wire set the actual production set does not look like this you know it's more uh pluggable smaller sets but we have a zillow wide wire set uh which has all the uh newers for all these common components and you know if you have a if you have some in direction uh for example you know like logger is used everywhere but we don't want every package to have to depend on a common logger package like server has its own local logger interface kafka has its own logger local logger interface which is shaped the same as the logger that we use throughout zillow but it's its own uh it's its own package its own uh type so we can tell wire hey when you need server.logger uh you can actually pass it the default log right so uh we create this wire set with all the initializer and uh now in your service you just have to say hey uh you know in my service i need somebody to give me a server factory i need somebody to give me a database provider i need to give me somebody to give me a kafka client uh who's that somebody that somebody's wire and you don't have to worry about how they're initialized you know whatever their dependency tree is somebody else will walk through that and you just have a way to create your server where you just handle your application specific routes uh create the server from the server factory handle your routes and wire you know handling your request is exactly the same right because all your dependencies that you are requiring are available in your my service and you can make your http request kafka requests and all that stuff uh and to initialize this server you just create a simple wire initializer right you say hey give me a server use the zillow-wide common set to populate all the dependencies and one additional dependency that my server has is this my service populate all its exported members right and most of the members of my service come from the zillow common set now instead of doing all this handwritten code this is all your server looks like you know your server has a simple newer and wire generates all this code for you super maintainable you know super lower low overhead for people to start creating new servers consistently another huge benefit that wire offers us not just why you know if you're using dependency injection or inversion of control in your application uh it makes uh testing your service end-to-end that much easier right your service your application just depends on this my service which has some of these types you know which are composed by some applications composition root you can substitute them whenever you want with mocks and then reason about your application your application code did not change it's 100 the same code that you would like to test comprehensively just your fringe dependencies which you don't care about you know you don't care about uh if postgres works or not you don't care about kafka works or not in your application you only care about if your business logic is working or not right so we along with a zillow-wide common set provide a zillow-wide common mock set you know where all these dependencies are substituted instead with mock implementation uh you know wherever appropriate we have interfaces for these types and we use mock gen to create mocks which you can set up in your test you know assert in your test verify behavior and stuff like that it just makes testing that much easier for example when you want to initialize a testable server you know you want to use your same server that you have been working with so that you can test your application logic but you just want to substitute you know where the database comes from so that you can pass in a sql mock database or an in-memory database you know or if you want to use an actual database in your cicd pipeline uh it does not matter you know the actual server doesn't care about who provided what database to it uh it just expects somebody to compose a database provider for uh the server similarly kafka can be mocked the database can be marked uh and the http client provider can be mocked so the unique you can intercept your outgoing http requests so you just tell wire hey when you are newing up my server this time instead use the common mock set don't use the production mock set or production set but use the production mock set and give me an instance of this server testable you know so that i can use this in my test and your server remains the same you know this is where your application code lives which you want to test and then you can write a comprehensive end to end starting from your transport to your dependencies uh single test case you know if you want if you are into code coverages this will give you 100 code coverage of your application uh you have intercepted your outgoing http requests here uh you have set up some expectations on your uh database and verified that they are happening similarly with kafka you set up your expectations and then verify that the right message was written to kafka and then you just use the you know http test package to create a request pass it to your actual application server the system under test and you have comprehensive end-to-end integration testing using wire as a dependency injection framework you know we're not saying dependency injection is the only way to write testable code but we find it to be least intrusive and a very efficient way to write a consistent way to write testable code and with wire you know it's very idiomatic it's very human readable it's very low overhead it's compile time there's no runtime magic here it's score that you can still reason about you know one big concern about dependence injection is i don't know where my stuff is happening from it's generated code which you can see and understand and follow uh so there's like almost no magic uh and since it's compile time you know if you have composition errors it will just tell them in your cicd pipeline or when you run the wire tool locally as i mentioned you are testing your actual server like for example here you are writing to http to your database to kafka you are verifying all these behaviors are working 100 consistently uh that's pretty much it you know those were most of the core components that we have at zillow to write a ghost services you know we i think we achieved some consistency all our services look and feel and behave the same way they are maintainable you know as i said common components change more often than we would like them to you know we have already changed how we do logging multiple times you know how we collect metrics from statsd to prometheus and you know we eventually want to go to open telemetry uh resiliency needs to be baked in for a production grade application you don't want hundreds and hundreds of developers to have to learn it the hard way right uh observability we talked a lot about and testability testability is huge you know if your test if your application is 100 uh testable through code uh you know that's amazing uh and i like to believe most of our services are well well tested and testable uh you know that that's it uh you know a big uh thank you to the go community you know all this work is based on a lot of these amazing packages you know so thank you to all the all the contributors to these packages

Channel: Gopher Academy

Views: 470

Keywords:

Id: 9Q1RMueVHAg

Length: 44min 15sec (2655 seconds)

Published: Fri Dec 17 2021