Microservices in Go • Matt Heath • GOTO 2016

so quick intro I'm at you can find me here on Twitter so if you've got any other questions just feel free to ping ping me and I'm I basically work as a back-end engineer I've worked specifically with go for about three four years now and at the moment I'm working with a company called Mons oh so Mons oh are a new starter we form the company February last year and basically we're trying to build a new kind of bank and that's kind of a bit of an interesting thing to do banking for me is kind of like this I don't know if anyone else has this kind of experience if you ever go into a bank branch that's super dated and certainly in the UK banking is is terrible as a mobile experience it's kind of come on the last couple of years but you just can't do the things you would expect to do from any other kind of internet product and you can book a taxi and it will arrive in two minutes you can book food and arrives in half an hour but I still can't like pay someone I've never paid before through any of my banking apps I have to like log into their website set them up as a new person and then I can somehow do that and it's just a terrible customer experience and a lot of the reason for that is because at their core banks have quite dated technology and this for example is the system/360 fun fact if you wrote COBOL or other things that run on this you can still run that on current IBM said series mainframes so that's great like you can run code from 50 years ago but it's not really ideal if you're trying to build new products you have these extremely large monolithic systems at the core of your bank and lots of banks use extremely modern technologies as a kind of veneer around these things but they can't change the core and as an example Barclays in the UK spent three billion pounds on IT infrastructure in 2014 who here has three billion pounds to spend on i.t anyone one put one guy cool so just think what you could build with three billion pounds like probably something a lot better than the current banking experience and we don't have three billion pounds not not by a long way and we're basically trying to build something that normal people would use people who spend all their time on their smartphone and they expect very quick interactions to sort out their finances they expect things to be automated and taken care kind of automatically so we started last year we give out super fun hot coral prepaid cards and we've basically been going through the process to become a fully regulated bank and that kind of happened a couple of months ago so these cars will convert into normal current accounts probably early next year um and these are connected to an app which is mobile first we kind of give you spending analysis we tell you how much you spend for example I've paid James's sent me some money here and actually shows his profile really basic things like icons of where I shopped and that's just not something that any current bank does um I don't know why it they just unable to and they're kind of constrained by their technology and all sorts of other fun things so you can do searching you can actually search with emojis so we we look up coffee it's a very important feature we can attach receipts I attach pictures of my brunch and other things like turn your card on and off with a single tap so if you lose it you can you know like disable it cool it's lost and then you can re-enable it magic and I mean this is obviously like a boon flag in our database right but technically an enum but it's I mean that's just something you can't do with any other Bank so if we're starting from scratch we would obviously have some form of application and all right the beginning we're startups so we should start off really really simple and inevitably we'll have some kind of applications and databases and over time this is going to get progressively larger and in our case we need to work with lots of distributed teams because we have to integrate with lots of different systems as soon as we expand internationally we have to integrate with lots of payment systems and these things get really complicated for example we have to have actual physical fibre delivered to physical data centers which is kind of a bit weird I just assumed you could put all of that stuff in Amazon these days but that's not a thing so eventually these things have become quite complicated and we potentially need more databases our application is very big we maybe had some searchings and caching various other features that are essential for a bank and inevitably and I mean the reason that we're kind of all in this room is that these things like you hit problems with them and you're actually hidden quite quickly and I've worked on a lot of kind of big projects where we just have one big application obviously we have to ship that we have release cycles and you just can't do anything and so this is kind of like my impression on that so right from the beginning we we decided we didn't want to do them and starting out by breaking our application apart into essentially a series of micro services at the beginning is kind of a risky prospect there's a number of things that you don't necessarily understand and you don't necessarily kind of break things upon the right way but we were able to start isolating those things so we started with a series of micro services fairly small these communicated via either synchronous or asynchronous message buses and the main kind of principles behind this were again like standard things that we've heard from people like Netflix Amazon Twitter are our services have a single responsibility they do one specific job very well in order to decide what they do we take our part our whole domain and bound the context so we have small chunks of our business logic which we can isolate into unique applications and then by doing that we can build very well-defined interfaces for example we can split apart transaction model and our transaction in my bank account shouldn't really need to know things about the merchant like if I buy something at a shop obviously that has a logo but the transaction doesn't need to know about that we can relate those separately and we can also because we're doing that we can apply different availability constraints to those so our transactional model needs to be extremely reliable but that kind of like merchant data if it doesn't work we could kind of provide four bytes to customers and we can kind of abstract that away and once we break these apart we then get really good composability so we can break our services break our application apart into reusable units reusable services and then actually we can build products much much quicker because we can reuse some of that functionality so when we started we had a number of different options and one of the things we we really wanted to look at was the kind of language we would build most these things in in our case we chose go so has anyone here used go yeah loads of people awesome so if anyone who hasn't used Go Go has a super cool really cute mascot this is the Go gopher who makes an appearance on pretty much every talk and the thing that I really like about Coe is that as a language it's extremely simple so there aren't a million ways that you can do things and go there's kind of like one way there are other ways but they're usually not easy ematic and it you're almost fighting against the language to do things in a different way however in addition to being statically typed which for a bank is quite useful we don't really want kind of we had type conversions and things obviously rounding money is quite dangerous ego also has a really comprehensive standard library so although the language itself is very simple there the language comes with a huge number of that you can use and as an example the HTTP server you can spin up a new server in about three lines of code and as of I think go 1.6 or 1.7 that is now automatically HTTP to server so all you'd have to do is take your code we compile it with the latest version ago and it will transparently upgrade your clients to HTTP 2 if they support it and that's just something that's built into the language like the the standard library has all these things built in on top of that the there is a number of really cool language features so one really big one in go is those extremely lightweight concurrency and by that I mean you can literally run any number of functions just concurrently with each other almost too easily to be honest so if we had a function for example which is going to handle a request this function call block so we'll execute them go off and we'll kind of come back at some point in the future hopefully we have some timeouts if we're handling requests and we'll assume that's dealt with if we want that to run concurrently say we are in a kind of loop that is reading off a socket and with we're passing requests off to something else we can literally just put go before it and that will now fire that off in a separate go routine and these long threads these are extremely lightweight almost like green threads if anyone's kind of used those a long time ago but they're extremely lightweight ways of running your Co concurrently and the reason they're so lightweight is when you far off a new go routine all you're really doing is allocating like a K on the stack it's like really really small and in go are the stacks automatically expand so you don't need to kind of worry about that the stock will start off really small is extremely lightweight and then as you're kind of stack grows then go or dynamically allocate memory to that so in practice you potentially have thousands tens of thousands even hundreds of thousands of these running and go has a really good scheduler built-in so the scheduler will basically run one operating system thread per call and then all dynamically scheduled in userspace these go routines across those threads and if they block on i/o then it will kind of put that to the side and it will schedule another go routine so that all of them can make progress if anyone wants to read about that is like really really interesting about probably a bit bit too much for right now but the really cool thing is like even Network IO isn't blocking so some of the talks I've seen before where we're using specific frameworks in hate say Java they're literally blocking on a thread while a request has gone to an external server now that's obviously like pretty poor design but in this case that doesn't happen like we run go routines we run potentially hundreds of thousands of them and every time we fire off a request to another server it actually goes into a separate i/o thread and all of the network is taken care of and it will just kind of pass that back to the go routine as soon as there's data to be consumed so given that we have all of these concurrent kind of almost threads running how do we then communicate between them so if we need to share memory if we need to kind of share values between these things inevitably we're going to have to put loads of mutexes and things around these and then go that's not the way that you do this so go advocate sharing by communicating between processes rather than sharing memory and this is kind of based on CSP if anyone's heard of that communicating sequential processes I think by Tony Hall in the 70s and channels provide a way that go routines messages between them and these can be either synchronous blocking calls or asynchronous so we can buffer them and put them in a queue and this again is something built into the language so we can literally pass messages between all our go routines and that allows us to do fun things like kind of funny out funding in messages we can run worker pools and all of these things are kind of enabled and used in the standard library just by being able to run go routines and having channels that communicate between them on top of that if you are building services and deploying them go is statically linked or leased it is if you pass in a specific number of billed flags so by default go will use Segoe which uses some of the go library of some of the C libraries available on your own machine but if you disable C go you can statically compiled all of the native go code and it will it will then run on any machine anywhere any machine with the same architecture without any runtime dependencies and this is really cool if you then want to very easily deploy binaries you can compile your whole go program to a single a single file single binary you can put that somewhere you can drop that onto a machine you can execute it and that's it you don't need a runtime you don't need to install loads of stuff on your machine so if you're kind of like want more immutable infrastructure then this is like a really cool way to do that so if you want to start with go as a kind of micro service framework there are a couple of things that you can use to do this so one of the big ones is go kits I don't know if anyone's heard of this a couple of people so peter borgin who is the kind of lead honest eye is doing I think a workshop later this week so if you're interested definitely go along and go kit provides a kind of framework that allows you to build services really quickly really simply and you can plug in all sorts of things like service discovery and tracing and kind of all the other kind of tooling things that you need out of the box and those are kind of all configurable in go kit there's another one called micro which basically is quite similar but it takes a kind of different philosophical approach so micro is more of a kind of ecosystem and as well as providing a framework that you can use to build services it also provides lots of other tools so there are specific discovery services that you can run id generation services all those kind of things so if you want to kind of build all those things into a system really quickly then microbe provides quite a few of those unfortunately neither of those existed when we start our company so we have one call month called time it's available MIT licensed on github and it's basically a really kind of really really simple framework so all it does is allows us to build kind of clients and servers that can communicate to each other and then we can configure them in a kind of standardized way so in our case we started off with potentially a number of services these are communicating to each other and the real question is how they communicate so in our case right at the beginning we needed something that kind of worked down the box really really quickly among way of kind of cheating is basically to put a message bus in the mail so if you use something I rub a McCue you essentially get a request queue and you can name these request queues and then you can have n number of instances of what you can to kind of consume off that named queue and that gives you essentially service discovery for free because you can just publish a message to a queue you don't actually need to know where the instances of a service on and it will be consumed at some point by a queue um it also kind of gives you kind of again cheating at load-balancing because the model here is a pull model so you have a queue of requests that are outstanding you potentially have a TTL on those within the queues so you don't fill the buffer and if we have one particularly performant instance and one which has a really noisy neighbors so we have really bad performance then those will just consume messages at different speeds and to be honest that got us quite a long way we started off in February and we've been progressively adding more more services to our infrastructure building out the features of our Bank core banking systems basically everything so 95% of the stuff that we built is from scratch and we've kind of starts off at obviously zero and we're now on about 140 or so services so the services that we run are quite small they do again one thing one thing well and we run a number of copies of those within our infrastructure for availability so when we started with this that was absolutely fine because the only language we used was go and I don't know if anyone's ever had the problem where you need to do something in a particular language and you don't have any libraries for it whatsoever some like everyone must have hit this problem so if you're using a particularly new language and go is kind of getting to the point now that it's quite mature you don't necessarily have libraries that cover every possible eventuality and as an example if you're integrating with financial infrastructure you basically have to use IBM MQ so who has used IBM MQ here loads of people or good who enjoys using IBM MQ here literally no one oh no one person okay sorry and in our case we which one to move fast as a startup so writing well there are libraries which you allow you to use those particular things but to do that we'd have to basically use the C bindings and compile them into our go binary which would be really terrible it also doesn't have the same level of functionality is not as well tested whereas using something like Java for that one specific service gives us much better flexibility and the problem we now have is if you have built a micro service architecture and you have n number of services that are communicating towards between each other you probably have some standardized way of doing this and if you've done it all in one language you've essentially built a quite large tool chain so just in that language which means introducing any other languages into your infrastructure becomes almost prohibitively expensive and actually the place that I worked out before munzo was halo where we moved from like a PHP Java kind of infrastructure there to one that was built in go with a couple of Java services for some specific things and the problem we had there was we ended up with 198 go services and two Java services so we have to somehow main maintain feature compatibility on these libraries even though only a tiny amount borrowing structure was was doing that so a quick way around map is to use a client and server the every language house which is obviously HTTP bit of a stupid things have gone all the way full circle but the reason that we did this was because right the beginning we could get all of those things like service discovery and load balancing free and we can write that in about three or four days whereas in order to jump in straight away with kind of basically HTTP based services you need to think about all of these kind of things so you need to be able to discover where your instances are which IPS they are ports if you're doing dns to resolve them or some other discovery mechanism you need to load balance between them you obviously need circuit breaking if some of them a slow will fail and the real question is how you do all of this and if you do all of that again in your own language your your again building this big client and server library that to all of these things so in our case we chose to simplify those quite significantly and we're actually using link addi in the middle to do quite a lot these things so as anyone's seen a pinnacle from Twitter I see it's written in Scala so it's not something we wanted to directly import into any of our services but linka D allows us to run that as essentially a standalone proxy locally and we can offload all of these things kind of into that proxy and because lots of instances are passing through this proxy it has a far better knowledge of what's going on in our infrastructure which leaves our services like really really simple and they can still have all of the kind of performance and flexibility we need but without having to maintain a huge amount of code to do all these kind of complicated things you need in a micro service architecture so to start with our the top of infrastructure we obviously have a load balance that comes all of our traffickers into we go into some kind of routing layer so this is just going to take a HTTP request decide where it's going to be routed to and drop it into our proxy and this is routed to an API service now these services are no different they essentially still just take HTTP but what we do is we define kind of our endpoints on our API by these particular types of services and that means for example we have a public API which has a web hook endpoint that allows you to register web hooks on your bank accounts and we can route all things that go to the web hooks route through our proxy into this go binary and then once we've done that we can kind of process that we're literally going to validate the request to send that on to kind of Authenticator maybe kind of actually create the web hook and register that for callbacks at some point in the future and these again have different databases and potentially those can kind of call out to any external provider literally is just going to be a bog-standard HTTP server that's going to process our request for us and the cool thing about this is because all I'm doing here is deploying a new section or API and we have this routing automatically if I want to deploy any new functionality which for infrastructure we don't touch any of the other services we can deploy services individually we can add new services we can add new endpoints on to our API without any impact whatsoever and the only realistically be the worst thing that possibly happen would be we deploy a number of instances and they consume all the CPU and memory on those particular servers and kind of those crash as a result and then we'd get replacements at some point so in go the way we do this is by defining a type which we call a service and all that's going to do is take any request so it's a function that takes a request and returns a response so if anyone's seen anything again like in pinnacle basically you you pass it kind of defining your service as a function you just pass in some requests get a response back and you allow kind of the the libraries to wrap all of these kind of filters around those and given that we can use a bog standard HTTP routing library so we can register routes on here and again each one of these so list register and D register is again just a function that takes a request returns response and it can do anything we need to do that so this stuff's like quite easy to do and go you can spin up HTTP server literally a few lines of code there's some really good routing libraries and you can just register your function just straight away so the main problem we then have as a bank is that we have to be really really reliable so it's all well and good having like fails and our services going down when it's just kind of the app that's affected but if it's our card payment infrastructure and no one can buy anything in the store then that's that's a really really big problem for us and this again is kind of the the reason that we started off with this kind of infrastructure is because we need that 24/7 availability we we can't add all the banks in the UK have like overnight batch processes or their systems turn off for like five hours overnight and and we can't really have that we we need our services to be on the whole time and that means even though we're running in Amazon or any other cloud provider we will inevitably have failure that we have to deal with so in our case we obviously everyone is using docker because it's so hot right now but in our case we can actually compile our go binary we can statically link it and we put it in a scratch container so we don't have anything inside that container at all there's not even a shell so if you had like a breakout like some kind of bug in our code there's no shell to actually execute code within that container and we can then run those on kubernetes so is anyone use kubernetes here cool a few people so from our perspective we actually start off with measures and marathon and the because right the beginning we didn't feel Cuba natives was really kind production level at that point and we had kind of mixed mixed experiences with that we found operationally it was quite difficult to run especially if you aren't really a kind of Java shop which were not so debugging it was quite difficult and overall I think the the kubernetes kind of design architectural is is significantly better having said that if you ever want to run it yourself on Amazon rather than using a self-hosted one to run that in are highly available mode is still quite difficult so there are a few blogs around them and it's kind of a trial by fire though but we've kind of tested our stuff quite a lot now and it's worked out quite well and the advantages we've seen from that is this is we did a kind of migration in August so what two months ago and this is the expenditure on our infrastructure so we started off with this red area which was our old infrastructure and we spun up on new of structure and then we kind of progressively shut things down and the reason that we've got great cost savings here is because we can pack our services much more densely and also we can use spare capacity for things like our compilation jobs so we used to have really big Jenkins machines which would like compile everything and push the resulting binaries up to s3 where as Cuban entries makes it really easy for us to just run those with a lower priority using spare capacity so we don't need to run any of those servers anymore so that's been a massive massive win for us so given that we have a number of instances these service we can use a combination of communities and linker D in our case to kind of resolve things like topology changes so if one of our services are explodes in a kind of shower of flames at some point fairly soon we'll get a replacement automatically because we've defined i3 replicas of our pods in communities and equally if we get slow or kind of transient errors linka d will kind of resolve these for us so internally is using exponentially weighted moving average load balancer which will detect failures become quite sensitive to latency and all these things kind of happen automatically and you can build these things quite easily and go using a host pool and there's really a couple of really good libraries from bitly that will do this for you but my experience is if you want to build like a micro service architecture you're kind of putting all those things in your services is kind of the thing you should probably try and avoid because again every time you want to add additional components you then need to build an exponentially weighted moving average load balancer in that service in our language and you just have like a huge kind of step before you want to you want to do that work and then the other thing we do um in RF structure is push as many things as possible to being asynchronous so there are certain things we have to do which are synchronous for example if if I buy something in a shop we obviously need to check I have enough money to actually be able to make that payment and in order for that to happen we have to make some synchronous call through and get a response back to the card Network and that doesn't have to be too quick but it does have to be a synchronous response however we use Kafka to publish events of everything that's happened in our infrastructure and for example if we have a service here we would have some kind of business event we'd publish that this has happened then any of elsinore infrastructure can consume that and then from these they can then make additional synchronous requests and if those fail the kind of queuing system will allow them to retry with some kind of exponential back-off the problem with any kind of infrastructure like this is inevitably you have no idea what's going on so I might know like this section and someone else in the company all know that section and this is some something that you have to deal with if you are making calls between different services so having well-defined interfaces which allow you to essentially read the documentation rather than speaking in depth to different teams is kind of really really useful for that the other thing that's really useful is various tooling so someone used kind of tracing systems like Zipkin know okay a couple of people so Zipkin allows you to trace a request as it goes all the way through your infrastructure which if you have a number of services they all kind of crossing a request boundary that that's extremely valuable because it's the only way you can really do bug things like this so go allows you to do this really easily so given a request that's kind of propagating through three services we'd start right at the top we in our API tag every request with a unique ID which is just a type for you it and then we pass that down through each request and kind of we basically marshal at once the wire send it as headers unmarshal it in the server and kind of pop it all the way down the stack and in go we can use the context package so go how use this used to be actually an external package and it's recently at 1.7 moved into the standard library so it's available built into the language and the context package is essentially providing something like thread-local variables so go doesn't have thread-local variables there aren't any threads and in a context it's basically an object that is immutable so it's thread safe and we can store values of any key and any value and that allows us to store our trace ID and pass it all the way through so we can pass that ID all the way down and then each one of these steps we can kind of send off instrumentation data and then we can aggregate all of that we actually use a tool called phosphor internally but Zipkin basically does the same thing and probably much better and then we get all of that data into something that is completely unintelligible like this but obviously this is intended for a machine consumption because we can take that and we can generate core graphs so this is an example of a kind being charged in the store so we have our API right at the top we have done this section that's synchronous which is kind of the minimum things we can do to charge someone's card we kind of need to look up the cars we have to record the transaction update the balance kind of things like that and then as a part of that we fire an asynchronous message on take Africa which is reliable and nothing else happens synchronously so in the app you have a list of things essentially on your bank account that happened and at this point that is not appeared and it will appear at some point because we have a durable queue but it's not essential that I appears immediately so we can kind of shunt that off to like an asynchronous processor and then at some point that will get picked up and we will take the transaction and we'll look up what the company was we'll try and get a logo for it see if it's already got one kind of like add as much metadata to that as possible and then we'll far another message asynchronously and then something picks that up and actually insert it into the customers feed in their app and then that again finds something is infamously and then at some point they get sent a push notification which looks something like this and the cool thing is here we have a coffee cup emoji because it's a Starbucks which is clearly an extremely important feature but the cool thing is that all of this stuff happens and you get this before the receipt prints on the machine even though all of these things have happened asynchronously but if anyone was broken your card would still work and you just get like a delayed message like 10 seconds later or worst case scenario if it's too late we wouldn't send the push notification but it would arrive and people can still use their card they can spend their money and they still have a very reliable system even though we have all this additional functionality we want to provide in actual well in actuality this is a massive lie what actually happens is that so there's lots of other things that happen and the other cool thing is each one of these things we can aggregate log lines using a logging library we have this trace ID we can write that on every log line we can pump that into analog occupation system and that means we get every log line for every one of these requests even though they're on different servers different services and we can tie all of those things back together and obviously expire them quite quickly because there's quite a lot of data and again you can use this kind of data to aggregate kind of maps of your systems and like really understand kind of how these things work so why should use go from my perspective go is a small simple and really easy language to learn so we found it really easy to be able to like hire people obviously hiring is quite difficult if it's language that not many people have learnt so having the ability to pick up really easy idiomatic code has been incredible for us and I don't think we've had anyone who's had any any problems of that at all on top of that go has excellent concurrency really good communication the networking stacks amazing so for micro services where essentially all you're doing is communicating over the network all the time that's kind of like a really really good thing to have there are inevitably some downsides so as I mentioned if you're integrating with kind of some specific technologies there might be not be libraries that particular one tested there have historically been problems with vending so if you have dependent libraries they obviously change over time and you need to kind of pull all of that source code into your tree so the way that we do that is we actually vendor every version into a single giant repository which is pretty huge but the advantages there is that we can vet those dependencies as they pulled in so we obviously have to do things like change control is as a financial company and we can vet those and we know that that is the code that's compiled and it's a deterministic build every time so should you start things with micro services probably quite a contentious thing I'd say the disadvantage of doing this is when we started building a company I knew nothing about banking and I probably still don't know very much about banking and if you don't know a huge amount about your your business that you're trying to build then it's very difficult to make those kind of delineations between the services that you want to build and you end up with interfaces that then change which then causes problems however if you do understand your infrastructure and you understand your kind of product that you're building really well then it's kind of a really easy thing to do you can easily pull these services out and pipe them across and go as a perfect language to do them and that's about it thank you very much so there are few questions yeah how do you transactionally commit events to Kafka so Catholic itself we run in cluster mode so you write to a partition try and remember exactly how the code works basically we when a request comes in we will write those into Kafka the credit card network is kind of an interesting one because it's essentially a two-phase commit we we will get we'll get an authorization message we will write that into our transaction database if that fails then we return failure back to the card Network if that succeeds but we still return failure then they kind of retry and if that still fails then two days later it essentially gets undone by the card network as far as I actually going to Kafka we we write everything to kind of partition so you have your key will basically hash to a specific partition and you write it straight into them so yeah there aren't enough to be failure modes I mean there always are in distributed systems but yeah Kafka kind of means once it's committed it's definitely replicated to multiple machines ok go by default has a very peculiar way of organizing source code files how do you structure your source code and why interesting and so go stores code in packages and essentially we would define a package as a folder with many files as long as they will have the same package name it doesn't really matter how they're broken up across those files but we usually try and kind of break up as an example if we have a service with multiple handlers so multiple kind of different requests that's going to handle each one of those will go in a different file within a handler package and then go provides a way to either export those or keep them internal within the package so we would export which you basically by giving them a capital letter at the beginning you export kind of a defined interface and the default thing is to basically keep everything encapsulated unless you specifically want to export like a kind of node interface that okay how do you define so next term sectional boundaries when multiple services are invoked within a single unit of work and what about rollback undo the service number four out of five fails within that unit of work so distributed transactions basically yeah that's that's like not really a thing so basically the way we do this specifically using Kafka for those things is Kafka is not really a queue it's a replicated kind of log and that means that we define define our messages very specifically so every message has a specific schema and we publish those so one worker will be consuming particular messages so if it can't process one of those messages it basically means the entire system is down and that consumer will kind of that consumer in a consumer group incalculable pause and kind of wait until we can then resume them so in the case of specific the only case that's really counts is where you're moving money atomically which is obviously something that's quite important and in that case we we do like kind of two-phase commit in the database and again if that fails our Cod processes upstream will send us a failure message and they retry that wall most indefinitely so all I could really say is like if there's anything you can ever do to avoid distributed transactions you should I'm sure you can by extremely expensive proprietary software for it but there's nothing really in the open source world that will do that for you so yeah we basically log transactions and then kind of retry them and alert if those have failed his database support good and go what are using yeah so database support is actually really good if you're using anything that's SQL based the standard library has an SQL driver you can then play in specific database drivers into that and it gives you a standardized interface essentially for like normal SQL databases we actually use Cassandra which is because we was we used Sandra if like 95% of the data that we use there are certain things you can't really use Cassandra for where you need kind of atomicity but Cassandra means that we can run a distributed database we run it with quorum reads and writes within a data center and we route request to the kind of correct place so that we can read and write a corn the support for Cassandra is okay and go I'd say most of the problems mostly outages we have a probably due to that and that's something that we are progressively improving but yeah a mixed bag can you show code and go with a micro service mm-hmm some samples uh as in some of my code yeah sure okay let's find something good not very sure wasn't scream come maybe not that whole thing go okay so let's use the web poker service as an example yeah I am just trying to find our first kill just close all the other things I do okay uh so Oh wrong screen yeah that one we have quite a lot services but oh I can't see this at all sorry everyone okay so if we have a look on our um web poke API that I mentioned a minute ago our - libraries open source so we basically in it with a few kind of known settings so all all go programs have an extremely small there we go all go programs have a func main and this is the entry point your entire program and in that all we really do is we define a service and then we run that service so that kind of like an internal library the kicks off HTTP server and then all our handlers are doing is each one of these let's find lists so if you want to list your web hooks which you can do by the public API we have registered get slash which would be slash web hooks flash and that is well this is kind of using a bit of a terrible compatibility layer probably the wrong one to have picked this is going to take in a HTTP request and some parameters and then it is going to check for an account ID it's going to return a validation error and it checked if you authenticated we also pull out your OAuth client ID and then this is the interesting part where we use protocol buffers anyone use protocol buffers few people okay so protocol buffers for us are just a way of defining essentially message request response and they allow you to compile that to any number of languages so it's really cool if you want to use request responses across a number of services and we just have our particular response type and we make a request if anyone wants to come in like specifically ask me this is probably not hugely useful on on this screen but I can walk you through kind of like a slightly more sample application what turn ative x' to go did you consider a good question so actually in our case um myself and a few other people worked at halo before so we moved from PHP and Java kind of originally kind of monolithic system to a service-oriented architecture that had kind of run full of services and then a few more services and then we rewrote that in a combination of going Java and actually as a result of um go was kind of our default choice we did we did consider Ruby for a really short period of time only because that was like super fast startup kind of get approached hype out but that would never have been a kind of long-term solution for a micro service architecture look you kind of suggest splitting your problem into micro services is hard is this not something then can be done continuously rather than upfront yes it is so if you're sorting all from scratch and you have a really really basic product then you probably only have a handful of services and you can probably understand that problem well enough to break the problem apart so in our case we obviously have some kind of known constants things like a bank account has transactions so we have a transaction service we have a number of bank accounts so we have an account service equally people have a profile with their name address and various other things and those things are very easily discernible the problem comes if you then realize you haven't quite broken apart in the right way and that's something that's much easier if you're doing a migration because you understand your problem domain much better than if you're building a problem like building a company entire from scratch um so it is a continuous process and actually on one of these slides I think in the the one we had no mail as long as there we go I'm sorry this is a good example where we we starts off with a kind of a slow growth and then we rebuilt some things and actually deleted a large section of our infrastructure and then that's kind of gone up and down and fluctuated as we've gone up and there have been many times where we've kind of refracted at one service into a couple or delete those and that's definitely a continuous process did you consider encapsulating that Java MQ and a go service we did however what we've actually done is encapsulated literally in something which will take in messages and call a specific other service which is go so we have a driver service that essentially connects to MQ and we don't pull the messages off and that's virtual it does so it's written in Scala and it's probably like 50 lines of code it's really really small and that means that all we're doing there is pulling messages off and dispatching them to another service which is an NGO which we then have all of our domains or chicken so it means it's like really nice and small and actually reusable so if we end up having to connect to multiple different providers but with different configuration we can actually just configure like this set of instances of that service to push data there and this one depreciate that okay that's it remember to rate and thank you man thank you very much you

Channel: GOTO Conferences

Views: 40,825

Rating: 4.900929 out of 5

Keywords: GOTO, GOTOcon, GOTO Conference, GOTO (Software Conference), Videos for Developers, Computer Science

Id: WiCru2zIWWs

Length: 50min 30sec (3030 seconds)

Published: Wed Oct 26 2016

Reddit Comments

[u/NikkoTheGreeko]

This was a good talk. We have also found HTTP to be a great way to communicate between services, even on the same server.