From REST to CQRS with Clojure, Kafka, & Datomic - Bobby Calderwood

thank you Alex and thank you Alex and Lynne and everyone ACOG attacked and involved for a great love coming out to this event this is sort of my first like big speak at the conference event I've done like little you know um sessions and meetups and stuff in the past and I have cange built up in my mind because so many of my like formative programming experiences and ideas happens sitting in the audience that kind so this is awesome it's great honor to be able to be care a great venue so my talk is on from rest to CTRs I think in the in the notes I know of Kafka I added Sam's a in there so it's in lispy parens and the idea here and this is like intentionally obnoxious the title of the stock right like oh yeah rest is rest is over it's totally done all the cool kids are doing something new right that was to get you to come to my talk we're gonna find that actually what the thing I built is is rather restful so we'll talk about that ah Who am I I'm a member of the tech fellows team and Capital One Kaplan is awesome it's becoming this like technology company and hiring a lot of really smart people and so I'm part of the fellows team that helps helped to do that helps guide that position very exciting times I formerly worked and have a very great honor of working on the product team at cognate X with rich and stew and all the smart people on that team it was a great honor and a really awesome opportunity to work on that so I helped build the atomic and certainly support it and help customers use it and be successful with it for a while so a lot of my thinking and the ideas from from this talk and you know really everything I've done since comes from day Tomic so you'll probably see that as the talk progresses and I've worked sort of flip-flopping back and forth in my career between little startups and really large enterprises and both have their unique set of challenges right startups you can do anything you want but you have no resources to do with big companies you have like tons of resources but you can only do like a yeah this narrow defined set of things so it's unique challenges and and it's interesting to try to to negotiate those things and I and I've come at it from both a business and a tech we'll roll my education was an engineering and economic so I'm really interested in how business problems and technical problems sort of interact and shape and guide it each other alright so how we're gonna proceed I'm gonna sort of define the problem I'm trying to solve because this is the closure conference I'm gonna talk I'm gonna define what I call essential rests contrast it slightly okay cue dress into a great degree with what I sort of call cred flavored rest I'm gonna introduce the ideas of seeker s and event sourcing talk about the design of the thing that I made demo the thing that I made hopefully that I'll go well and try to assess the benefits and trade-offs of this thing that I made how I'm not gonna proceed this is not as I said at the beginning of the talk does not a repudiation of rest there's like so many branti blog posts out there on the one hand defending rest on the other hand lambasted for its shortcomings this is not that kind of talk in fact the thing that I made like I said is quite restful and has all of the sort of good characteristics of rest the things that I call it essential rest so we'll talk about that this topic of secure s has kind of come up quite a bit in the closure community recently someone made a yuppie chef I guess is a company made a thing that was secure RS so it's been in in the the mind share of the community so I also want to make it clear that this is what I'm proposing is not the only possible implementation of CQRS this is just sort of what fit my design and problem constraints so here's the goal you'll notice this is not a very specific well scoped goal right this is like a big thing how do you provide these valuable informational and transactional services as a company to your customers potentially lots of customers how do you do it with a great user experience how do you do it securely and in compliance with regulations you know some industries are more heavily regulated than others and how can you do it in such a way that it's easy to on the one hand experiment and do try crazy things so that you can try to find things that work and satisfy your customers needs but on the other hand make it easy to operate and stable and easy to maintain stuff so I'm gonna solve all of these problems in this talk get ready so the problem one of the problems that that I'm facing now and that I've been thinking about is organizational scale imagine you work at a large organization you've got lots of product teams both sort of horizontal product teams that drape across a lot of different lines of business and provide shared services and then individual sort of product development within those lines of business that try to do things that you know make money and attract customers and all that stuff right now in a lot of organizations including my own two point something that faces the customer and provides value to that customer involves a lot of work it's like this whole big stack thing and tiers and stuff a lot of code and you have to you know build this UI in like whatever UI platform of choices and then you've got to build all these ap HTTP API is to like service that thing which leads to HTTP a API proliferation right someone came up with a great idea of like a service-oriented architecture micro services let's just build a bajillion of these things I don't make it easier right you end up with a lot of these things and a lot of these things face outward to you know power the customer facing thingies and some of them face inward to power internal thingies and some of them face your partners isn't only they can have access to to integrate with other organizations and that's that sort of things so there's just a lot of these things there's hundreds in the sort of financial industry there's hundreds and hundreds of these api's in a given organization I there's lots of and DevOps goes a long way towards sort of taming this complexity and helping you get these things into production fast and help you continuously deliver them so the bugs get fixed quickly and all that stuff but that really only goes so far part of the complexity is the fact of the proliferation right the other problem with this is in this proliferation all these API teams who reinvent wheels all the time right every thing that you put into production in this sort of you know micro service you rest you kind of world requires a big stack of really complex things you have to solve these hard problems over no again distributed systems problems arounds failures and retries and timeouts and and you know all that stuff that's hard that's our problem right you've got Netflix out there with hystrix and and those types of techniques to try to like mitigate this hardness but like it's still hard you gotta solve security problems and authentication authorization problems we've got to solve audit and regulatory comprar blebs you've got to have a database to store your stuff you've got a automate like the the continuous delivery of all this stuff so everyone who deploys an API into production has to solve all those problems everyone that employs a UI in the production has to solve a certain set of problems how do you talk to all these different endpoints they're all subtly different even if you use things like swagger and you're using a common framework across all these things they're still gonna be different all the semantics are a little bit different they've all got their own little own mini language for accessing this data and for submitting these transactions right so your UI has to like okay what are all the things we have to talk to okay let's write like a little client for how to interact with each one of these things and then only and only then after we figured out all this hard stuff can we actually get to the important business of user experience and making sure the animations are smooth and making sure that it is the thing the customer wants and all that so all the UI teams have to solve a common set of problems as well and library and code reuse only go so far right that that tends to be the sort of the default mechanism that we reach for when we want to not solve these problems over and over again is hey let's just make a library for that it looks like a framework for that right it's like a knee-jerk reaction and and you know we do this internally a Capital One I'm sure at other organizations it's like hey let's come up with a encapsulated way to communicate with these api's or to do our authentication authorization schemes and so forth but again that only goes so far and has the unfortunate side-effect of layering on all kinds of complexity all right I'm gonna generate some code I've got more code I'm gonna add a library oh now I've got like more things to trace through the stack trace when something goes wrong you know more more of those layers from Stu's TOC of when something goes wrong where could the problem be see if you've made an internal framework it's probably there right it's not just it's not just in your a in your application code that's not the only likely place for bugs now right now you've got all this other stuff that you have to potentially debug so my ideal state what am i shooting for here I want to make it really easy in a large organization to simplify UI development so I can go really fast so you can try 50 experiments a year on different UIs I try to meet your customers needs what not the cool try 50 things a year I want to simplify HTTP API development by not doing it so much anymore deleted codes debug code right so let's let's just do less of this I also want to sort of share responsibility across this large organization it's a hypothetical large organization such that the central and sort of horizontal team can manage the the the reinvented wheels problem they can solve the hard problems once and all the vertical teams that are actually making you know products that go into production like UIs and business logic and stuff they can use those shared services they don't have to solve those hard problems over and over and over again and I think most importantly and we're gonna talk about this is Lorde ad ad nauseam in this talk I want to record a really clear picture of my interactions with my customers so that I know what they were intending to do that is like sacrosanct right that's all that matters in a service is what your customer is trying to do and whether or not you help them do it right so I want to record a clear picture of my interactions with my customers so that I can make sure that I'm doing it right so that in you know highly regulated environment you can make sure those interactions are within the bounds of the law and eventually we'll talk about this maybe towards the end to be able to replay that series of actions into a new set of business logic right wouldn't it be cool if things change or if you find that your customer has a unique set of constraints or more attributes that maybe they should have had this other set of business logic applied to them as they were interacting with you but you didn't know it at the time so now you've got to go back and try that so talk about that a little bit so what is essential rests I made this bold claim that there like these two different halves of rest what does that mean what are the good parts of rest rest is awesome I love rest I've read you know the cover page and abstract of relief elding's dissertation so it makes me an expert in rest I'm going to tell you all about it no really I have read a lot of that thing it's just really long but it gives really clear communication semantics for big distributed systems in a low trust and low coordination context right when you're on the big wide internet I don't want to have to you know coordinated with every person out there that's putting stuff on the web right that would be terrible first of all I don't trust them because they're probably a malicious hacker second of all I don't want to have to coordinate with them cuz I don't know them and it's really hard to talk to people so let's uh you know I'm a software developer I don't want to talk of people so this loose coupling provided by rest the proxying and caching over this big network with sort of unreliable and unknown Network hops between places and stuff it's really great I don't have to know what you're sending me I we can sort of negotiate it but about it at the time you make your request and sometimes you can even do really cool things like conditional requests but not very often right who really uses that feature another good part its self documenting and describing this makes are really nice I think this is one of the primary drivers of the adoption of rest as developers like it it's really easy but like okay I'm gonna click around an HTTP interface figure out how it works get a sense for what the entities and actions are and then I'm gonna like you know try to interact programmatically there's even things like swagger and stuff the kind of document a little and it helps you sort of build and learn these these resources quickly and sometimes people even use hyper state that's the medium is the engine of state transfer but not usually right that's why we have swagger you know do you really use that thing it's also data oriented which is good right closure data good requests and responses or data sort of after you parse it and do all kinds of crazy things to it but you also get these really nice differ representation formats different languages different encodings and we can kind of negotiate about it sometimes there's like even real content negotiation but again not usually a lot of the tools don't even sort of ignore brush over that part of the rest what's content negotiation kudos to some of the library's enclosure that do a really good job of content negotiation liberator in particular is the thing that I used some kudos how are the bad parts of this essential rest uh yeah the person with a hammer everything looks like a nail right if I'm gonna write software man it is gonna have an HTTP API or else it's not worth writing right not every piece of software needs to have this HTTP API right it doesn't have to be driven by hypermedia the this knee-jerk reaction has led to sort of the pop culture sort of adoption of rest and these implementations are often bad which lead lists of this so I think rest started out really great and a lot of people adopted it for the right reasons and then people started adopting it because you know that was what was cool there is definitely a fashion component to our industry as much as as loath as we are to admit that we sometimes follow fashion there's all these tools that make it super easy so like do this crud flavored rest thingy this immutable you know they've already decided for me how the URLs are gonna be formatted and the sort of semantics to put in post and and all the hard stuff I don't want to read that that dissertation it's all my tools can figure that out for me right so I can just generate some stuff and and now I'm rest awesome it's widely supported and understood there's lots of docs blog posts videos and stuff it's easy to heart higher for provides nice guidance for younger developers right you just generate code you know rails new whatever add this whole thing and it just works right you also end up with tools like swagger and stuff for specifying and validating these things so that's good I mean there's some good tooling and and mindshare built up around this sort of crud flavored rest bits and all the cool kids are doing it right this is a good thing right right it's it's agile with a big a hey it's easy to get something into production even if it's like whatever poorly specified and we don't know really what the semantics are of our domain but it was so easy to get in production we don't need to know that let's defer understanding our domain until like our customers start submitting bugs it's better than soap so that's a good enough reason to use it right it's better than the immediate predecessor so let's all adopt it right and then you know there's this whole culture of DevOps II micro-service cloudy stuff and it all seems too loosely intersect with rest so if I want to adopt this other good stuff I have to adopt rest right and if it's good enough for companies like Netflix Twitter Facebook and all the people who really do this right who really deeply understand rest that I had a fundamental way then it's good enough for my company that probably doesn't right so it's hard to start to tease apart these things sometimes which leads us to the bad parts all the cool kids are doing it that's a bad part rest is over someone on the internet says because these people have flooded into our our perfect rest afar Ian temple and have defiled it's all of the unwashed ER here and then they don't know what they're doing just kidding that's not really one of the bad parts some of the bad parts that I'm you know in my view and I think maybe some of us in the closure community would share is that this takes us back into the tar pit this notion of this crud rest especially the you part and maybe even the D part in in cred this takes us back into the end of the tar pit we use these beautiful tools in the closure community and some other communities like these functional languages with referential transparency so we can trace our values all the way through think no global state well very little global state we usually the immutable data structures even some immutable databases right this is all to get rid of complexity right this is complexity is the enemy well happy brontosaurus he's not in the tar pit in order to implement this thin veneer over mutable objects in the large right so we've we've cleared out the complexity from our process space like inside our JVMs and we pushed it back out into the semantics of our larger systems this is not an improvement this is this is a retrogression sad brontosaurus so we're back into the star pit where we're we're managing state all the time right these tools and I understand this has nothing to do with like the dissertation the essentials of rest but the tools as crud flavored rest that has grown up around rest promotes mutability promotes thinking of these resources as like these tiny mini databases where there's a thing and I can read it or I can create a new one of it and then I can like bake on it and just like update over the top of it how am I gonna update over the top of it I don't know put maybe patch what is that thing who knows how that works right all of these things promote this in place mutability that's bad that's bad don't do that I would submit to you the crud flavored rest is the kingdom of nouns for distributed systems right we end up yeah I see I hate these heads nodding I'm not the only one that thinks this yes the crud notion makes us Jam every problem domain into this really narrow box of like these creations and reads and updates of these of these little resources not every problem domain is like that some are right if you're making like a content management system or something and maybe maybe you're just like slinging representations these things around that's fine but for a lot of problem domains we actually like do much richer things than create a thing or update its attributes right in my problem domain we're like moving money around this is important stuff and it's got these really rich semantics that that are really difficult to encapsulate in this crud notion so it creates this impedance mismatch for example I want to a pay my credit card bill or B post a new bill payment sub resource to the credit card top-level resource no no you've obscured your problem you have added incidental complexity when you do that it also burdens your clients with resource orchestration right when you've decomposed these really rich interactions into these just like creations and updates of all these different disparate resources you've placed the burden of orchestrating that rich interaction onto your clients which is usually bad alright we want our clients to be thinking in terms of the user and to be user advocates thinking way that the user thinks but now we've pushed all of our crazy back-end bound cents up into the front end so that they have to think in terms of the way we think on the backend and not in terms of the way the user thinks that is a bad thing yeah how do we transaction and modify several different things at once conditional requests and then finally this sort of kingdom of nouns effect of the crud pattern has the same consequence as object orientation in that it proliferates endpoints and proliferate semantics around these verbs and proliferate dispatch on all these crazy attributes of the request the same way that object orientation proliferate state access patterns right instead of having a whole bunch of operations that act on a very few data structures now you have a whole bunch of data structures all of these resources and a few actions that sort of you have to learn for each one of these things that's bat the following criticisms I understand apply to the out-of-the-box implementations and and a really a lot of the prior criticisms of this crud flavor dress applied to the sort of out of box implementations that come with these tools for for developing these things and now I'm gonna talk about the operational characteristics of the things that you build a lot of times with these tools you're deploying the single artifact that has this internal router that routes to a bunch of different resources and all those things sort of version and change and deploy and whatever together scale together that's bad because they're all complected together they're all tied up even if they're unrelated scaling an individual resource right the customer resource versus the Trant you know the pay my credit card bill resource scaling them independently becomes hard right now I've got it maybe refactor or some code or L it's like deploy a whole bunch of versions of these things that I only intend to you know load balance certain requests to certain clusters of these machines it places a lot of the burden of load balancing and scaling these things onto operations which is bad and scaling reads separately from writes is hard so some of your resources will be more frequently access than others and certainly and a lot of problem domains reads and writes have a unequal operational load you're gonna be reading much more frequently then you're going to be writing but if they're at the same end point and ones just a put and ones just they get you it's really hard to scale those things independently so that you have a whole bunch of read nodes handling your your hire read traffic and a fewer write notes and handling your write traffic so of course they're operational things you can do to work around this right you slap Zul in front of it and reroute these requests that not seem but that place is the burden of the complexity that you created and development onto your ops team finally oknot finally there's two more sort of criticisms it's hard to extend to integrate these things so i'm not gonna talk too much about this but when you've got this into your application you've got the front which is the api HTTP api and then you've got the back which is the database and everything that happens in between is sort of this black box throw lights degree of course they're all ways to work around that but for most of these tools out of the box you can either integrate out at the front-end with speaking HTTP or you can integrate in the back end in the database integrating of HTTP layer is slow and sort of piecemeal and you know imagine doing like a big MapReduce job over over all your resources using the HTTP interface seems slow because you're speaking HTTP it's gonna hammer the availability of your endpoints which is bad what's your other option to integrate and extend this thing after the fact out of the database what are you gonna go ETL that's horrible don't do that pulling stuff out of the database and putting it someplace else no don't do that that's bad or you know haven't forbade you actually like use that database and read your transactional data to do some analysis right that can hurt the availability of database anyway so that's not a good thing finally and probably most importantly this crud flavored rest obscures the story of what your customers are trying to do you lose the story you don't understand in domain terms a clear interaction with your customer first they tried to pay their credit card bill and then they hit the the not found page or the error page and then they you know try it again and then they left the site you don't have that you've got this crazy convoluted impedance mismatch thing of like where when these requests happen and you have to go back and piece this together with like software archaeology and never done a spun clog mining to try to figure out what your customer is trying to do and they bailed out or alrighty that's bad that's bad we lose the story of what our customers are doing so what is secure a sentiment sourcing CQRS is simply an architectural style in which the read paths in your application and the right paths in your application are split handled by separate models separate code pads that's really all it is it's very simple thing in HTTP terms maybe this means we have like two or maybe three endpoints we've got like a command endpoint we've got a query endpoint and maybe we have like a website excuse me maybe have like a WebSocket thingy to subscribe to new reads as they come you know because the request response nature of HTTP so insecure s there's sort of a lot of baggage if you like google CQRS out there and your seats you're probably gonna come across a lot of like passionate advocates of secure s and finally detractors of CQRS but a lot of them are really talking about sort of secure s in the small secure RS is sort of an object-oriented thingy for how I structure my classes and inheritance and how these classes interact inside of my local process space right it's very much in the small secure house I think at that level is actually a bad thing because you're gonna end up with more classes right for every customer class that you would have had in your old object-oriented model now you have like a read customer class and a write customer class so you end up with proliferation and complexity it looks bad but cigarettes on the large just what I'm talking about this is sort of as the architectural style for your big distributed system out there in the large and that actually reduces things right you go from this crazy proliferation of a bajillion D endpoints down to three which is good I've just oriented CQRS as I sort of discussed gets all tangled up with these other concepts domain driven design and aspect oriented programming and a few years ago kind of 2011 2012 there's a lot of writing done in the object or in the community about this stuff this is not what I'm talking about when I talk about CQRS at this level we know better than that anyway it's functional programmers we know the language of the system we know that we want these immutable values to flow gracefully through our systems and isolate and delay these side-effects until the end so again I I submit to you without any evidence or argument that a lot of us are a lot of those arguments against seeker s out there on the Internet I don't apply to us secure SV HTTP might look something like this right I've got these three endpoints there's like some business logic and stuff and then you store stuff in the database well I might use this I submit for simplicity's sake operational simplicity and agility and as I hope we'll be able to talk to if I don't go over climb and implementation simplicity and agility event sourcing is an idea that sort of closely related CQRS a lot of times they're sort of all wrapped up together as well I think they're a good match men sourcing is modeling and calculating a system state based on all the events that have ever happened to that system right you sort of reduce over that set of events and you figure out given some initial state where in this state transition space we are which is good it's easier to do in CRS because you're getting all of the events through your command interface right so you can just siphon those things off storm someplace you maybe update an index or a materialized view or a database or whatever and then that will service your query endpoints this is architecture look familiar to anyone this looks sort of vaguely familiar yeah I'm gonna appeal to Authority well I simultaneously in one fell swoop one fell stroke can give gratitude and credit where it's due and also unabashedly appeal to Authority they Tomic what CTRs before plus event sourcing like before was cool maybe it's not cool yet I don't know but yeah a lot of the goodness that comes from day Tomic comes from the fact that you have these separate read and write paths and you know rights go through this thing where you can control them in their atomic and written down so this log and then on your read paths you're reading from these from these indexes that the log you know after the log gets written they update so that's cool there's a lot of like goodness that comes from this type of system you also get to be able to reason about this this logical notion of time right if your event sourcing all the events that come in each event is like a tick on the clock right you get this notion of time in your system and I submit to you anytime that you have like state transitions over time in your application you should probably use event sourcing right that's why I reach for diatomic every time I like want to write an app because that gives me some level of event sourcing for free now it's a little bit different because the atomic is like data interaction that that's like have a data level of event sourcing I'm gonna submit in my sort of broader design here that there's another level of interaction that's important and semantically meaningful which is the customers intent level right that's the domain they're expressing their attended their intent and domain terms whereas the atomic sort of captures that notion in data terms so another reason is that it's easy to extend now that you have this sort of event piece of your application now you can integrate at the event piece you don't have to integrate out of the HTTP level you don't have to integrate at the database level you can integrate at the event level and capture and siphon off these events pipe them to some other process some analytic process or something right in the middle of that and you've got this really handy hook for extending and integrating your systems why might I prefer crud flavored rest over this new thingy that I'm proposing perhaps if you already have external SLA or whatever that you know customers are already using your thing stick with it right it's not it's not worth the big rewrite in most cases if your problem domain really is resource oriented then model it is a resource oriented thing right do do cred that's cool but for a lot of problem domains it's just not alright this is my possible design it says design here in this very authoritative font but this is just one possible design for how to architect a distributed system using these these principles so here we have these kind of different layers you have two clients out on the front the web services layer this sort of command blog thingy that captures all the events that come in the front door some business object that reacts to those events and then a place where data level interactions take place down at the data materialized views level I chose Kafka for our like spine the the command log piece and Samsa to react to those things they go really well together also it is a spoiler alert the remainder of this talk is sort of a thin wrapper around Martin Clements talk on D structuring the database great talk gave it a strange loop couple years ago a lot of those ideas influenced a lot of this but this is closure we can write thin wrappers and other people's work right that's ethical I'll do that around this talk so I could have chosen other technologies here I could have chosen lambda Ben van Griff and adam hunters talk on lambda right that can react to stuff could use that instead of Sam's a here could use maybe Kinesis or something instead of copy I so there's different ways we can handle these things with different ekta the important thing is the architecture what do I mean when I say command I'm gonna create a command in my proposed architecture here it's a 2-tuple of a an action and a data payload what is a query it's a big common query cuz it's awesome I don't want to make anything new I'm just gonna allow my customers to you know you know submit data log queries to this query endpoint I'll sanitize it I'll authorize it and make sure they can only see the data they want to see but you know that's a great quarry language I'm not gonna mess with that updates are a WebSocket message that shows you know consequences that have occurred very much in the same schema as actions but this is you know to inform the customers or the consumers of that thing so the client library its job is to sort of fetch the initial state from the query endpoint to subscribe to a stream of updates to keep their state up to date and then to submit commands via the command interface right I just post to this command interface and I can like make new things happen in the system and these things aren't like these little tiny atomic resource updates these are rich semantic things pay credit card bill create customer launch missiles or whatever right like it's the you can you can submit these things in domain terms so these commands very much should be able to model your domain pretty closely I would submit to you much more closely than this sort of post to this and get to that and all that so just to be obnoxious this is the priority of the closure script client before the these three endpoints that I'd submit to you right there's just three public functions send a thing send the command queried some data subscribe to a channel of updates that are coming from the server so it's a really a tiny thing that's the only reason I flashes up here I don't read that don't read the code I just flash up there to show you that this is a really small thing this takes a lot of the burden that used to exist for the UI's and client implementers and has really reduced its scope and complexity also I think I think he's gonna play really well with them next I was very gratified watching David stock I had read the documentation on him next and play well together but after his talk yesterday I'm pretty convinced that this architecture that I'm proposing would be a really awesome back-end for own next I'll next already has this sort of secure se notion of query paths and transaction paths on the front end so I think hooking those things together would be really really useful the web services their responsibilities are sort of take care of all the edge concerns load balancing monitoring service health all those sort of operational characteristics it needs to authenticate the incoming requests to make sure we know who this person is making this request it needs to validate for syntax the incoming request body so like a command and make sure that command conforms to the unknown schema for a known command in the system and then authorize that requested action right so I can pull the action part out of the command body and say is this person authorized to do this thing great if not throw in a 403 Vinod authenticated the room 401 right this is all rest stuff and then the only other thing that it really does on the command path is just write stuff down a Kafka it's this really thin thing it does a whole bunch of edge concerns to make sure you know cuz we're getting requests from the scary internet but then the only thing that it really does just write stuff down to Kafka forgets about it doesn't need to know any business logic it doesn't you know anything right and then a response to queries and then it pushes updates out to the clients very very thin web service layer this is the you know again don't read the code but this is deaf from liberator a rest and we handle things in this post method we've got two paths here we've got an asynchronous post which literally just writes down to Kafka and then returns a two oh two like yep we heard what you said good there's you also have an option to pass a synchronous parameter because you want to sink in request for some reason and then we listen to Kafka and when it either succeeds or fails then we notify you with of 201 or you know 422 if things didn't go well so again this is all still rest this is the entirety of the meaningful parts of the query interface I just like send the query today Tomic cool again it's also rest but now instead of the rest trying to map to your domain semantics rest is really just describing your communication semantics like this is how you interact with my thing it's not trying to you know Jam the crud and all that stuff close to your domain semantics the command logs just Kafka it stores things I'm writing things down as fresh in because Kafka just like stores binary blobs of stuff impressions awesome so I just write things down as fresh into Kafka and then Samsa is listening you know consuming these streams I've got three streams there's a pending transaction stream you know accepted transaction stream and a failed transaction stream so the web services layer writes down to the pending transaction stream a business logic piece reacts to that and then either says this thing succeeded or failed according to the whatever business rules you have and probably in the course of that business rules processing is going to write the consequence of that down to diatomic or wherever that's all it really does so there's sort of three different layers of things here in the business logic piece there's the canonical business logic the transaction processing piece there's the you know that's the thing that I just described but then there's arbitrary actions this is where you can extend your system to do new stuff you can listen to a list of failed transactions or a list you know a stream of accepted transactions and then do something with that update a dashboard for the boss or pipe it off into some big data analysis process or whatever like you can just extend and enrich your system without touching or even caring about the implementation of like the canonical part of the business logic you just can extend stuff it's pretty cool and then oh yeah Sam's a job is a really small thing it's you implement a Java interface with one method it's really a very small thing that's one of the things that drew me that too sam says it's really easy every time the new event comes across this stream from Kafka it calls this one method pretty cool and then finally like data gets stored in places that place can be sort of the canonical place where things are stored the system of record I think maybe Kafka is also a system of record but it can go to other places write some data Lake great great phrase it can go to some materialized view to like a cache or something to make queries faster all right I'm gonna demonstrate this because I actually like built this thing which is pretty cool here is the HTTP interface for the thing that I wrote we're going to send a successful command to the command interface here and that command is to create a customer with the following data attributes submit that thing yeah it worked I got back a 201 I know if you can see that down here yeah whatever it's sent back a 200 on promise and then sure enough we've got a new transaction there 10:39 we can also send that synchronously and it'll return us a 201 it'll just work cool so that 201 like went all the way through the pipeline and went down to Kafka Samms reacted to it put it back on the stream got read back by my web service and it's pretty snappy it's it's fast enough right the default case is asynchronous so that customers stuff can keep working you can optimistically update and then check back later to see if that thing succeeded or failed query interface same same kind of deal I'm gonna grab a very simple query here you know submitted this is very much like the rest interface in DES Tomic sense query through I can you know provide a as of T that I want a query ad or I can make it a history database I can submit arguments to this thing to a parameterised query or whatever but yeah it just sort of works they thought make is awesome and I'm actually not gonna have time for this piece of it but here I've got repla interactions for using the client library I wrote that I showed you on the screen here for doing essentially the same things like I was gonna show you that the updates come through on the web socket and all that stuff but just trust me it works because I'm running out of time all right so let's talk about the positive consequences of these things I'm not going to talk about the negative consequences because I think it's awesome I know actually that's part of my future is to try to explore the trade-offs associated with this design but for now everything's pretty good first of all it allocates responsibilities properly in the sort of organizational sort of business concern way the central can like the horizontal teams can manage the shared service can manage the canonical database can manage like an environment for running the business logic but they don't have to write the business logic they write and maintain and deploy the web services tear and Kafka but that's it so they're handling sort of the hard problems that you want to solve once the the edge team like the vertical teams that are writing stuff that actually goes in front customers are just writing you eyes and just writing Samms of jobs and just maybe doing some like big data processing or something on the backend for analytics but all that stuff doesn't touch any of the HTTP level stuff they're not writing web services it's a really simple way to allocate that responsibility properly no more web api proliferation that's a good thing there's also no OLTP ola p split right all this is like you can do analysis on live data in this system it's really cool forget about that you know that divide that's existed since relational databases became a thing it's implies a business logic you're just writing the samsu job it's easy and simple it's implies client development because the client has like a library now that they can abstract away all of the complexities of the communication with the service and they just communicating in domain terms via these commands and in query terms again um next I think is a great candidate here for writing UIs and most importantly to me this reifies customer intent which we have a stored record caught one of cool things about Kafka's you can just say hey when should this topic expire never and it'll keep everything around forever so all of the events that come into Kafka are there forever to be replayed played into new business logic like v2 of the business logic you can just like subscribe to the stream of events from the beginning of time and like pipe that into like v2 of your business logic and hot-swap over without ever touching your Web Services stack no more Bluegreen deployments right this is cool and then this is useful in certain contexts like mine for regulation audit compliance and and that sort of thing being able to replay a customer's actions into a new set of business logic for for various reasons is important so next what I'm going to do I'm going to justbe hosi by making it a little better and you know trying to try to make it work I'm gonna weigh the trade-offs I'm sure there's some downsides to this thing I gotta figure out what they are and I also want to sort of explore the broader idea here that I've started to it started to crystallize in my mind which is I think databases and the the sale and adoption of databases subtly influence the psychology of technical management where we started to view our organizations as databases right we're just this big data repository and stuff happens and then we change ourselves and stuff happens and we change ourselves but I think we should recast our notion of how our companies and services and products work into sort of reactors to these streams of events as events come in from the universe we do something about those events maybe we generate some new streams of events they flow out into the universe yeah we probably update some data and stuff and that's cool but really these these companies and services are reactions to to these streams of events do credit where it lies a lot of these ideas came out of talks with Timmy Walden rich Hickey rich one day that said to me try to make something more data-driven than like a rest thingy that was cool and then certainly day Tomic again certainly Martin klepon and Ando next I'm really excited to see how this could this idea could jive with ohm next as being the client front-end thank you many questions

Channel: ClojureTV

Views: 25,451

Rating: 4.9302325 out of 5

Keywords: clojure, Datomic, programming

Id: qDNPQo9UmJA

Length: 44min 24sec (2664 seconds)

Published: Wed Nov 18 2015

Reddit Comments

[u/Frozenlock]

I hope Bobby will make his demo application available somewhere.

[u/youhavethewrong]

I was kind of skeptical about this one because of the sensational title, but it ended up being a pretty good talk.