Building Event Driven Services with Apache Kafka and Kafka Streams by Ben Stopford

[Music] there we go welcome to the final day of death ox my name's Ben Stopford and today I'm going to be talking to you about building event-driven services with patchy Kafka and Kafka streams so I work at a company called confluent anyone heard of confluent well we've got a fair few that's pretty good what about a patch Kafka hands up for a patch of Kafka got a few more for that okay awesome so um confluent is the company this sits behind a patch in Africa so we basically provide like support and there's a sort of open source ecosystem behind this open source streaming platform and if you haven't come across Africa before we'll be talking a bit about it today and that's essentially what it is it's a open source reaming platform and it's probably most well-known for the distributed log which is a sort of messaging system a very scaleable messaging system that sits at the start at the center of this platform so I actually work as an engineer for most of the time on Apache core Apache Kefka core the log and I've done some stuff like beta balancing I think the latest version of the replication protocol stuff like that um so if you want to ask me very technical questions you can come up at the end I'll be very happy to answer those but this talk is really about thinking about how we apply these technologies to building services distributed business applications and really just using these stream flow processing tools so when we build them when we build software just kind of taking a step back when we build software we have a couple of different masters one of these is kind of very intuitive and very tangible it's the idea that we just build features right you are you are users they want stuff done so we oblige and we build software which does things that they want whether it's putting information on screens or letting people buy iPads or whatever it might be but there's software engineers we actually have another hat that we were and that other hat is really this idea that we're thinking about fact that our software needs to last right so why do you modularize your code why do you implement unit tests why do you do continuous integration or continuous delivery all these things are practices which you apply because you know your software needs to last and software lasting doesn't mean it sits on some file system in github it means you can evolve it in real time and actually when we build systems this property that a system needs to be able to evolve it's actually as important in system design as it is in software engineering so this this concept of evolution is very important and hopefully through this talk we'll see that one of the things that taking this kind of event driven model helps you with the other one is actually scale which is hopefully be pretty apparent but one the other thing that it helps you with is his ability to evolve so streaming platforms they come from a particular background and streaming platform looks something like this sort of a canonical use case so you'll have I'm sure a mobile phone in your pocket that mobile phone will have a number of applications on it which will be sending packets back to spurious servers for the applications that you have running it also have an operating system so that will be sending data back to let's say Apple or Google or whatever phone you have and these datasets get very large and they're quite small messages right so when you open an application it sends a little message to Apple or a message to Google and you can't have lots and obviously lots of this implement these messages passing in real-time so these datasets get pretty large so you think you're thinking of the order of about 100 terabytes a day of what are very small messages and a streaming platform lets you collect that information in Apache Kafka which I said is this kind of messaging system and then it also lets you process that data and the reason that's important to process this in a streaming way is because the data set is so large so in a streaming way um you perform some kind of computation and you pull perform that computation incrementally so rather than you might with this or a database or a batch oriented model you can sort of collect all the data you'd let it build up over time and then at some point you sort of run a query across the whole thing where streaming's about kind of continuously computing a result and when you have these big fast data sets this is sort of a necessity so in this kind of architecture we'd have we'd land this large data set into Kafka we process it with this stream processing layer which sits in the middle and then we typically put it into some serving layer and the reason we need the serving layer is that everything that's happening in the streaming platform is asynchronous all right some messages are flying around there's actually typically multiple stages so you'll have multiple steps and actually messages will move between different instances of the streaming engine and at some point you want to bridge that kind of asynchronous divide with a sort of more synchronous divide where I can run queries to get to get the results that you need so that's why you put it in a serving now it's a few things here there's a this high throughput messaging system the stream processing layer is optimized for scale and also this idea of data locality so one of the things that stream processing does to enable the scaling is it has some primitives for bringing data inside each engine and that's some that's very important because it stops the whole engine coupling around a single data source and then finally if you're using kafka streams which is the streaming API which just ships with Apache caf-co it's just literally one of the api's that you can program against and you can build a streaming layer with that that's literally just a Java app right so it's a it's an API public static void main and you just start at new K streams instance and then you start playing with these streams that live in Kafka so that's just like a little overview of what a streaming platform looks like so let's try and get an idea of what stream processing is because it's not always the most intuitive concept so imagine that we have a set of authorization attempts and that so we've got various people around the world are using their credit cards and we're interested we're interested in looking for fraudulent fraudulent attempts so somebody using a credit card at the same credit card many times in a short period of time it's a very very simple streaming example so if for authorization attempts coming in people using their card and possible fraud coming out so we can express this as a query in a language called K sequel which is as you can probably guess if you're familiar with sequel quite similar to sequel and this is one of the api's over apache kapha over apache kafka streaming platform so what's this actually doing well firstly we're taking in a constant stream of authorization attempts and then we're creating an output string so these are potential fraudulent transactions and we're defining that as selecting the credit card number with a count over a 5-minute window and this idea of a window is really important we're going to talk about it a little bit more like but this is how we break an infinite stream up into something that we can reason about I guess it's over a five minute tumbling window and then we're going to group by the card number and then filter with a having Clause account more than three so within a five minute window and that window will basically move in time we're going to constantly look for any credit card which which has had more than three attempts so streaming is really this idea of having a tool set and it's actually quite a rich tool set which she lets you manipulate data in flight so this is a kind of very simple that was a very simple example we just had one stream in and one stream coming out but you can actually join multiple streams together you can then you can offer those you can perform computation on those for example Maps reduces all of the sort of things you see in the sort of Java streams API though all of those those api's are available and you can kind of add any functionality that you wish send emails call out to other servers perform very complicated machine learning algorithms okay so that was a bit about streaming what's just got to do with business applications let's take a look at this well increasingly today we tend to build ecosystems right so we're whether we're a startup for startup we're quite likely to start with a single monolithic application but pretty soon we're probably going to want to bring in some other kind of application or scale house over multiple machines for some reason likewise if we work in a big company we probably have a load of applications already so we have to think about how we're going to join those things together and over the years there have been a number of different patterns for doing this today micro services are pretty popular before then there was service-oriented architectures and event-driven architectures but these are they're all they're all these also slightly different patterns for working out how to join a set two independent processes together so that they further some common goal so in this instance this is a sort of microservices example where we have we've divided our domain up into a bunch of services which have single responsibilities so the problem with service architectures tends to be around data so let's take em I'm going to use an online retail example quite a lot during this particular talk um so imagine you know Amazon or you're buying iPads from Apple or something like that so let's say we have this this kind of architecture where we have with segregated responsibilities so we have orders information inside the orders service we have customer information inside the customer service maybe the product catalog lives inside the stock service or maybe it has its own service that suffice to say that we have all these data sets and we locate them inside the services that own them that makes a lot of sense so um and we can obviously we may be exposed at rest interface on each of these services so that we can get at this data but the kind of important point is that these data sets the customers orders the product catalog we're going to need them in a whole range of different services so when we look at the shipping service which is the service which is going to take an iPad from the warehouse and get it to your door it's going to need access to these data sets to do its work it needs to know customer information so that it knows where to ship your parcel to etc so how do we share data between services so if we simplify this model a little bit we have simple web application and it's talking to an order service and a customer service so if we buy an iPad we make a call to the office order service and that would talk to the shipping service which could also talk to the customer service and shipping service would get the iPad to our door so simplest way to do this is with rest or GRP CEO or whatever your response style protocol of choice and the web server would so we buy an iPad and we the order would be submitted to the order service and then the order service would have some probably some service discovery mechanism which allowed it to work out where the shipping service wasn't it make a call to the shipping service to say you need to send this to this person and the shipping service would then make a call to the customer service to work out where to send it to what's the address of this particular user so get customer and then off it goes so this is a this is okay this'll works quite well a couple of things make this slightly more difficult in practice and that is when these cool graphs get very complex so notably is when the call graphs actually add several layers of depth so once we have services which effectively chained to one another and one of the main reasons for that is that each service become very dependent on the performance of another so for example when I submit an iPad I actually just want that to come back as quickly as possible I want to know that it's gonna be shipped to me but uh but I don't that service doesn't actually need to be responsible for making a call to the shipping service like if the shipping service not isn't running you kind of the order service has got to like buffer and keep trying until the shipping services back up again so the way people get around that problem is using an event-driven model and the vendor of models are slightly different and there's this sort of pros and cons here you can use you can mix these two models together but I think it's just important to understand what the trade-offs are so in the event driven model um the order will be submitted in exactly the same way right we're still going to make some kind of rest call a WebSocket or something and remit code the order service but this time the order service doesn't actually know anything about any other services all it does is it raises an event order created again it sends that to a messaging system and I've used caf-co because I work on Kafka you could use any messaging system you wanted at this point it doesn't matter it's a suffice to say some kind of publish publish subscribe besting system will work wonderfully here so the order gets created and the shipping service would subscribe to this order and it could end response start the shipping process likewise we have this sort of customer service the customer service can do the same thing so when a customer updates their information it creates an event and that event gets sent to the broker and the broker transmits it around but there's something interesting about this idea of an event and that is that it has a kind of duality so on one side an event is a notification it's a call for action a call that somebody needs to do something and it's actually decoupling the two services so the order service has no idea who's going to use the order created vent event the shipping service is the one that subscribes to that event so we have this concept of notification the second point is that the event itself includes data right the event is the order it's a fact so the second part of events is that there's this other side of this reality is that it's a built in ability for us to move data around the architecture there's a mechanism for replicating data because the data is actually inside the event so in the customer case you may not update I mean one was the last time you changed your address probably the last time you moved house that might have been a couple of years ago so there's not gonna be any correlation between the events coming for your order service or the events coming from the customer service so in an event-driven model you would store that event in a database so you basically have a you'd use that's as a replication string so have these two notions of events and two ways that we can use them so on one side we can literally just use them for notification and the decoupling properties that come with that actually and scalability properties that come with advocates were moving into this asynchronous world so if we just used it for notification we would the order service would submit the event but actually the shipping service would make a synchronous call to the customer service to work out where it needs to send each parcel but it also we can use this other side we can use the depth data replication size where we're actually optimizing for data locality all right so in this model customer information is replicated through events into a database in the shipping service so effectively using it almost to tie these kind of two databases together and that means that the customer sorry the shipping service is going to make that query locally so there's kind of these two hats there's this notification hat and there's this replication hat and we kind of we can use whichever ones we wish so events actually provide an important key to building scalable ecosystems because they provide notification they decouple services they also move you into this asynchronous world and then in a synchronous world you can run many things concurrently streaming is the toolset for dealing with these events on the move so streaming is all about dealing with these streams of events that flow through a system slicing and dicing them moving basa to different services and performing computations efficiently across multiple machines so it's great to take a quick look at Kafka and so category as I said is a streaming platform and it has a few different components and going to touch on what what each of these are and then after that we're gonna look at how we actually build up an event-driven Services ecosystem so at the core is a look a distributed log and the distributed log is a type of messaging system so you put a message in it goes to a topic other people can subscribe subscribe to that topic topic could be orders so the interesting thing about kefka's distributed log is has no bottlenecks so if we have sort of producing services on the left so we've got three of those we've got four consuming services and we've got a path go which will will say is running over whatever five machines or something it would be anything between typically three although you can run at one but typically three or hundreds it's about as big as kefka fosters get so the important point here is that you can scale this at any layer there is no bottleneck anywhere within the system so you write data in and when you write a message it's actually routed to a particular shard and we call them partitions and that partition will live on a particular machine and within that partition you have very strong ordering guarantees so that partition is actually when you read it on the consuming side what actually happens is a partition is allocated to a single consumer and then there affects these partitions are effectively handed out and that's how consuming services read so we can scale this architecture out linearly at any single layer so this would be this in this case these consuming services would be spreading the data set across them and it's actually that is this architecture which allows you to deal with those very high throughput workloads that we talked at about right at the start of the talk when we're talking about Apple and Google and stuff so each child is a queue and computer because humours can share load across themselves by effectively because they're and have they have individual partitions shards assigned to them and if one of those goes down it just automatically recovers and that's actually in every single layer so what one of the the side effects of this is that your ordering is based on partition that's your ordering guarantee so if we were processing orders and an order has a lifecycle like it's created and then it becomes validated and then it maybe it gets shipped we want to make sure that there was an ordering guarantee across that lifecycle so what we do is we just assign that the order ID to the key and that way would always be routed to the same partition um what if we need global ordering well if we need though broad ring we actually just create a topic which has a single partition that means it would only run on one machine so it would go back to being just like a normal message broker like TIBCO or RabbitMQ or whatever so we can always go back to this global ordering property if we want we just sacrifice an element of scalability but we get all of them of the resiliency properties so we can use this in the service architecture to do a few things we can load balance our services so we can happily have multiple instances of a service running at one time and in if one of these services fails there'll be a very short pause and it will data will be routed to the surviving service effectively load balance as well as provide fault tolerance and this fault tolerance actually exists not only in the service layer but also inside the broker itself so the Kafka broker replicates data across different machines so that if any single machine fails and it will just automatically recover and what this means is that you can run a system always-on so for example Kafka came from LinkedIn originally that's where it was built and LinkedIn uses unsurprisingly quite a lot of Kafka as a result and they don't take their services down so these clusters run all the time if they need to do a release then they simply do a kind of rolling bounce where they stop a machine upgrade it start the new machine start the old machine and the whole thing kind of cycles across so there are a couple of other properties which we need to focus on one is this idea that we can rewind and replay the log so the log is actually unlike a traditional messaging system it's literally an append-only data structure and you can keep very large data sets in Khafre as a result so it's not uncommon to have topics with hundreds of terabytes of data inside them and that means you can keep data effectively indefinitely so if you're it's not just you can use it not just as a messaging system for moving data around you can actually use it as a storage system and then rewind back to the start and replay data in the order that achieved it was created so this gives us very interesting and useful primitive for backing a service ecosystem and then finally I want to turn this too much but there's this I have to mention it there's this thing called a compacted log and this is a effectively a log so log is immutable by default every single message will be kept sometimes you actually don't want to have every single message and the main reason that you don't want to have every single message is because you want to move that data around and when you want to move it around you kind of want it to be as small as possible so you can compact it based on a key and a compacted log ends up looking like a table in a database effectively so that's the log that sort of sits at the core of the platform nexor connectors I'm not going to spend much time on these these are pretty intuitive idea we have a connector and it allows us to pull data from a database and turn what is off typically beautiful state you turn that into an event stream of every single change that the database made and then we can put that through Kafka and then on the other side we can take that event stream and put it back into it into a database and there's a bunch of kind of off-the-shelf connectors on the confluent site which connect to pretty much any database that you can imagine the nice thing about these more contemporary connectors particularly the ones on the left the ones that suck data out of a database is they use this thing called CDC which is change data capture and that's a very efficient way of getting data out of a database because it doesn't sort of go in the front door it basically latches in on the transaction lock the bin log which sits underneath the database journals have everything that it does and it turns that into an adventuring so doesn't put too much operational overhead on the database so finally I'm always talk about streaming engine and so Kafka streams is a database engine for data in flight so we saw this kind of query earlier so very sequel like do group bys actually if we using the java api we have all of these sort of math transform pete functions as well so we can do anything that java can do so we can run these compute these queries continually i can join datasets together we can apply windows to allow us to reason about infinite streams we can then transform into views so in a services model that often means turning some message format into our own domain model and then applying filters where clauses aggregates on top of that but this kind of window this idea of windows is quite important so imagine I have like an emailing service then and I'm working in an asynchronous world so I want to send an email to a user when their order when they create an order but also that order is paid so you can imagine there's going to be some kind of order event and some kind of payment event and you might think there would be some some that one of these would turn up before the other in an asynchronous world there's no way for you to know which of these two are going to turn up first because could be delayed on the network there could be a whole host of reasons why the order might turn that turn up first or the payment might turn up first so you need to join these two things together and this is kind of what um so this is what a window kind of helps you with strictly speaking I should say there are two concepts here there's windows and retention they're very slightly different but for the purposes of this talk I'm going to treat them as the same so we can use a window to allow us to join these two things together and this helps us deal with this asynchronous world now I've put a five minute buffer there so what the streaming engines actually can have to do is it's gonna have to offer these events for five minutes and work out whether or not when a new event comes in whether or not there was a correlating event which is going to join together and then send an email and Kefka streams actually will buffer this data on to disk and there's a reason for that which is that you can often want to have quite long windows this like this only there of them offering can be extended to something that's actually slightly more powerful which is a table so let's say we want to do something like that example we have before where we want to join we what we have orders and payments but we also want to join customer information so orders and payments they're correlated in in real time right they're not going to happen in exactly the same time we don't know which order they're going to turn up in but we do know they're gonna have to note about the same time we have no idea when the customer last updated its customer information but what we can do here is we can basically create a window it's actually retention which goes on forever that starts from offset zero and remember Africa is a storage system as well as a messaging system so we can actually store that customer information in Kafka and what this means is that we're effectively going to join these two offered streams orders and payments with a table of customer information and we're going to do that inside the email of service I said this is all happening inside this API so we can blend these concepts of streams and tables and this sum this leads to this quite important consequence that were able to optimize for data locality we're able to push data to where it's needed so in a stable stream processing engine we can have a stream might have a compacted two stream and we can push data to a service so this will be the kafka streams API on the left and we can join multiple streams together and we can join tables as well these complete result sets like all of the customers that we have and one of the main things that a streaming platform does is it optimizes to solve this particular problem it optimizes the data locality and helps you scale this problem where you need more retention and ensures that you don't have to spend a long time rebuilding these views if things go wrong so if we want to store the customers if the customers don't fit on one machine we can spread them across multiple machines to scale this topology out and this is all happening this all happen in kind of outer site inside the API literally when you write this code you literally just write a stream you open a stream much like that sequel statement we had at the start and you just process that stream if you create a table it will just offer that table locally it will load the data if it hasn't already got it etc etc so streaming is about a couple of things it's about processing the data incrementally so rather than putting it into a database in computing it in batch it's about constantly processing beta as it happens as it arrives constantly reacting to the world and that's that sort of notification side of event remember we're talking about events earlier all right so let's have this notification hat but then we also have that replication hat that fact that an event is really a fact and that fact is moving can be moved from one service to another so stream processing is also about optimizing for moving data to where it needs to be quickly so that's the streaming platform we didn't really talk about the producer and the consumer in the top they are just very simple API is for getting data in and out of Kafka okay so we talked a little bit about streaming we talked a little bit about streaming platforms let's talk a little bit about how we build services so this in this section we've basically gonna go through ten steps and involve a little micro services ecosystem and based on this technology so the first thing is to start simple and keep responsibility hate responsibility for the past and evolve forwards so quite often you will be starting with some kind of legacy application or some kind of monolith and you'll typically evolve for words we might split out an order service which we talk to and as we talk to as we discussed before um one way we can do this is to use events rather than talking to services and the reason we do that is because of this queue ality we want to leverage this duality of events the fact that they provide decoupling by a notification but also a mechanism for us to localize state when we need to so instead of talking directly to services we Rea's events in this case the web server will journal and order received events just like the previous example and it be picked up by the order service now we need to obviously evolve for workflow forwards right an order can go through a variety of different states so in this model the user would request the iPad the order would be received it'll be picked up by the order service and then the order service would validate the order and then raise a new event called order validated and in this model anyone who obviously wants to listen to that can listen to that so weird we totally decoupling ourselves this is actually got a name it's called receiver driven flow control and it basically equates to plug ability in your architecture so next thing we can do is we can use the connect API and that's cool change data capture thing to evolve away from legacy this is a very common thing to do so let's say in our database we have the product catalog and we actually need the product catalog to be available both is a data set but also as an event string then we can literally just attach it onto the bottom of this legacy database and we can suck out the product catalog anytime it gets updated inside the legacy system it loves to get pushed into this event stream and we can actually leave that they drink after if we want to so for is about making use of schemas so typically you don't have to do this but it is a good idea and that is when we move data around whenever we particularly do fire-and-forget messaging we want to kind of wrap it in some form of schema and the reason to do this is mainly backwards compatibility so if I'm the order service and I'm subs bribing to events I kind of want to know that they're always going to be compatible with my software so we can apply a schema to do that so confluent provides a schema registry um we use Avro if you don't want to use Avro if you want to use protobuf or something else you're more than welcome it's just that schema registry won't work at this time anyway the Avro is actually very cool but the key point is that you need some mechanism whether it's Avro will protobuf or whatever to allow you to reason about whether or not data whether or not they just changed on the wire so that your des service knows that he's always going to be backwardly compatible that's really for me what a schema is that's the most important property of registry gives you next we can use the single rider principle so remember we had this order went from received and then I went to validated now we can imagine it goes to order completed so what we kind of want to do ideally is locate all of the logic that trend that takes a particular entity forwards and we want to locate that in a single service so that it is the only thing that can write an order progress an order forwards like most of these things we don't have to do this but this gives us a couple of lot of nice properties well the first one is this it allows us to reason about consistency in a single place and the second one is that actually makes things like schema migration when you when you need to change schemas much easier so in this in this instance we're effectively have a restful service up at the end and we're submitting an order at t1 and you can imagine it kind of comes through here the order service is the single is has a single writer but all of these other services are effectively working at different points in time and because there were here at different points in time if we isolate the consistency concern fact that that order has a state change in time in a single service that we can manage this in a distributed architecture so a couple of reasons for doing this creates a glocal consistency point because in a big distributed architecture we often don't have a single consistency point so the good example of this is say you're globally replicated let's imagine your services exist across multiple geographies it's like you probably don't want to write back to a single database in one region and it also makes schema upgrades easier if this one Service owns how that particular thing is mutated in time so six is we can store data inside the log so messaging a nice thing about categories its messaging that remembers we install these very large data sets so we can let you just keep a data set inside kathcart if we want to so for example we might keep the data set of customer information so that when we start a new service let's say we're going to build some new service we've got one here yeah there we go so let's say we build a new service which does repricing my sister it's gonna work out whether or not we need to change the price of iPads as people buy them with the supply and demand if that needs the product catalog we can build that service really quickly because the data set both the product data set lives inside care but also obviously this stream of orders is something that we can subscribe to so this kind of makes the architecture very pluggable so we can use it as this kind of single source of truth and this really relates back to this point that in a services architecture where you have many different services it's actually not the data that sits inside the service that matters so much it's the data that they share because that's what everyone else is using so if we do store datasets inside the log then we have to think about how we query them Ike how do you query a look right it doesn't have a queryable interface literally all you can do is listen to events as they arrive or you can rewind and read the whole thing or read from a certain offset and the way we go about this is by moving data to code so that should sound a bit strange hopefully because it is a slightly strange thing to do for many many years we have moves code to data mine databases are all about the idea that we put our data inside a database and then we move our code our sequel to our database and it does the computation in reality there are many use cases where you actually do want to move data to code and again this is to do with locality so a good example of this is if you build like a web application you're probably going to have you're going to use a bunch of caching over your data to your caching is actually doing that it's providing a mechanism to effectively move data that your code and then kind of keep it there so a lot of things we're going to talking about here are really evolutions of this idea of caching but caching is effectively more declarative we can be much more specific about where data ends up in different services within our architecture so let's say for example um we have stock this is the information which is available inside the warehouse we might decide to move that data set from Kafka or and originally the whatever stock service into the order service so actually gonna move the whole data set can you think about moving a data set is a bit like caching but we're actually gonna take the whole thing one optimization is that we typically only take the data that we need so let's say why does the if the orders service needs stock information it's because it needs to know what how many iPads there are available in the warehouse so to know how many iPads are are in the warehouse I actually only need two pieces of information I need to know the product identifier which is probably like a long or a string or something and I need to know another maybe long integer which is the amount of data the memaris by pairs that I have so we actually take these relatively small data sets actually the process of doing this is a lot of it is about optimizing for the data that we move so when we move data remove data to a service right this idea of pushing data around the network the ecosystem we have to be a little bit careful about a couple of things first one is that some obviously Africa scales really well you get incredible flip through but it runs that were limited so network doesn't tend to be such a bottleneck so much certainly not for like business systems it is a bottleneck in big data systems but even even in those actually indexing tends to be the thing that takes the time so whenever you create an index is the act of creating that index that tends to dominate your performance that tends to be your your point with the point where your bottleneck so we can use the log instead of a database so this this is a so in this instance we would basically take these four serve stocks we might have them stock service we have reserve stocks and we're actually going to effectively mutate data this is this is event sourcing and we're going to journal when an order comes in we're going to reserve that iPad so that nobody else can take that like that and we can store that back in Kafka so finally just on this on this idea of data movement I said streaming platforms optimized for data movement and they do that as they have standby replicas this checkpoints and compacted topics these are three mechanisms which just kind of help you deal with this problem of not having to load large data sets on startup so almost at the end now so we've got the nine is we can you try all of these things together with transactions so that kind of that kind of event sourcing model we're actually mutating state inside the order service we're updating our reserved iPads but actually create a number of events you have no you know event coming in you'd commit an offset you validate the order you'd reserve the iPad and kapha provides transactional guarantee that wraps all of that if you're writing to Kafka right you will call out to a database or send an email and all bets are off so just like a database transaction orgastic guarantees only live inside the service so we can wrap a transaction around all that stuff so finally we can bridge this sink async divide so this is a kind of more a full arethere system of the thing that we've been talking about so on the left hand side we have a post and a gets simple rest service behind that we have a load balancer and we have a set of three instances of the order service and this is bridging this synchronous Devon I want to post an order and I want to get an order back and it's bridging it into this ecosystem which is all event-driven so the fraud service the order details service the inventory service these things are all running highly available and they're all event-driven so there's a sort of flow of events which is club flowing around that way each we can just scale this out to as many services as we need they're all run concurrently and highly available they'll evolve this this these basically do validation and then back in that orders view on the bill or some left hand side that's what we're basically we'll start to build up a view of the various different orders which wish to get the get request will respond to we can do this all in a kind of Numb in a sort of blocking way which allows us to read our own rights if we want to so in this example you can find the code for this online it's on the confident Kefka streams examples I think it's a reference at the end and have this is a sort of sample microservices ecosystem which evolves in order for words and then we can scale this out to larger ecosystems where we have many different services and we're able to push data into different services we can even scale this to global ecosystems so if we're spanning data from so we can move data from York London Tokyo that's kind of one of the beauty that's kind of the beauty of an event-driven model is it doesn't really care where the data is whether or not it's you know on another in another geography on a handheld device on a mobile phone etc so good architecture to my mind is actually not so much about drawing something on a whiteboard it's actually very hard to draw a good architecture on a whiteboard at least it's hard to reason about a good architecture on a whiteboard because it's a very static thing and actually the essence of good architecture to my mind is this idea that systems need to build evolve in time just like we know the software needs to evolve in time and we do things like test and you know motorized our code of code and use continuous integration so it's really important that we actually applied this idea of making an architecture evolutionary request Riven is absolutely something that we need we'll always need the request driven paradigm particularly for dealing with you eyes and and things like that there's many times when that's the right protocol to use but we also want to think about leveraging this cue ality of events the fact that we have notification and beta replication this ability to push notify as an event but also that ability to move data from one service to another and streaming provides this kind of unique mechanism for dealing with these data sets as they move in real time I am adding that into an API which you can embed inside your services or if you're off the JVM you use case equal so event-driven services broadcast events retain them in the log evolve the events training with simple streaming functions right so he's literally just gonna attach a function which takes an event and moves it forward rich is it makes it become something new recasting the event stream interviews so this is how we because when we move into that asynchronous model we need to get back to the synchros model again how do we get back again we recast those event back interviews which allow us to then query them so that's that's basically CQRS and we do this with a confidence streaming platform and essentially all I had to say thank you very much my time is up and if anyone has any questions do you have time for questions now or you don't want to see if anyone has any questions I'd be very happy to answer them if you'd like to ask a question afterwards I'll be around for the rest of what's left of today so I'll be happy to answer questions about this or anything else have co-related and there's also if any of these in these ideas were interested you were interesting check out the blog series actually a new one came out today which includes that example which I didn't have enough time to really go into with the the streaming order service and it has references to all the code so you can download it and play with it if you want to so yeah anyone got any questions hey that's one here [Applause] yeah so the question was is the order only guaranteed within a partition yes the answer to that question is yes so if you want global ordering you use a single partition but that will be fully replicated and fault tolerance but your your throughput will be limited to one machine and if anyone else has questions just come down and ask Thanks [Music]

Channel: Devoxx

Views: 47,872

Keywords: DV17, Devoxx

Id: p9wcx3aTjuo

Length: 53min 3sec (3183 seconds)

Published: Sat Nov 11 2017