Reliably Save State & Publish Events (Outbox Pattern)

when you start using messaging you'll notice that your application often times updates the database and then needs to publish an event or message to the message broker the problem is because the message broker and database are two different resources you're going to need a distributed transaction here's how you can avoid the distributed transaction but still atomically saved to the database and publish a message hey everybody it's derek martin from codopinion.com if you're new to my channel i post videos on software architecture and design and net so if you're into those topics make sure to subscribe so i'm going to show this kind of in a diagram situation and then i'll jump into some code to show it as well is that you have your client and a database your primary database wherever you're storing your data that could be a relational database a document store doesn't really matter but the point being in your code you're going to be making that statement to save some data or change some data to the database and then a secret step altogether is where you need to publish that message to your broker or to your message broker to a queue the problem is because these are two different operations what happens if you were able to save to the database but then for whatever reason in code some you had a failure where you couldn't actually publish the message this could be because for many different reasons let's say that the queue is unavailable for whatever reason or more likely you have a bug in your code that you get some exception that you run into for whatever reason god forbid it's the lovely null reference exception maybe there's some edge case where the data that you're putting in the message there's a failure there so you've saved to the database but you are unable to publish the message now this is a problem because if you have other services for whatever reason that need to know about that that event that message that's occurred because you made some state change they're not going to know about it so things like oftentimes are used for cash invalidation it's not gonna get invalidated you're gonna have stale data or even worse if you have long-running processes where you need an event to occur to kind of kick off the next step it's never gonna occur so this is the problem this is why you need a distributed transaction so you can atomically say these are all one thing but we're going to avoid the distributed transaction altogether i'm going to show you a different method without box pattern that a lot of libraries implement that you can just start using to illustrate how the outbox pattern works is that you're ultimately going to be publishing your event and your state change together in the same transact transaction as your primary database so what that means is when you save something to your database is step one and then step two is you're actually when you're publishing your message and this is um how the api works is gonna be dependent on the library messaging library you're using so if you're using brighter or mass transit or end service bus etc they'll all implement this in the api slightly different but ultimately what's happening is that the message that you publish that you're seemingly publishing to the broker is actually getting saved to the database a part of the same transaction as your state change so then what happens is the the library in a separate process or thread will then fetch from the database the messages that haven't actually been published yet and then it will send that to your message broker or your queue once that succeeds and it was able to do so it goes back to the databasing either deleting records or updating records that again it manages about okay these are the messages that have been published what this does it allows you to publish a message technically and save your state change in one operation one atomic transaction to the database so when you're publishing your message it's not immediately going to the message broker you're actually saving it to the database with inside the same transaction as your normal state change so i'm in the loosely coupled monolith project that i have i'm in a branch that i have that's using cap which is a messaging library that supports the outbox pattern so here's code that i haven't implemented using it and you can see again the issue is that we're making some state change i'm just using any framework here where i'm saving an order to the database and then the problem is i'm publishing this message but once i save this and it's been saved to the database if there's a failure at any point here on out i'm not going to be able to publish a message if there's some exception or this this particular method call fails whatever the case may be it's not going to happen if there's a failure after this point after we actually saved it so the way cap works is again it saves everything to the same database and you can do this by creating a transaction so i'm going to say transaction and then i'm going to do db context database and the way cap works is that it has a extension method for a begin transaction where you can pass it the publisher so i'm going to give it the publisher here and then let's wrap this in it so really what this is done now and then when i do my transaction commit so really what i've added is adding the transaction to the database and telling it to use the publisher to use that transaction instead of it publishing directly to in my case i'm using rabbidmq it's not going to publish directly to rabbitmq it's actually going to save a record to the database all right so i'm going to just run this and then i'll show you the dashboard for cap so you can see the message is publishing as well as i will show you rabbitmq so you can see that they're actually getting published there and then i'll also show you the database itself that you can see how these tables get created by cap but again it's gonna be dependent on the library that you're using so i'm just gonna jump into postman i'm gonna fire off um one request i'm actually hitting my break point here so i'll just step through it i'll remove it and then we'll fire off a bunch more so we're starting our transaction we're saving changes but we're in a transaction we're publishing our message which is not gone cap yet we're committing our transition transactions all going to the database so i'm going to fire off a few more here just so we can see that they're occurring so now if i jump over to rabbitmq here we go here's the messages that are getting picked up right now and then if i jump over to cap cap has a little ui dashboard and i can see these are the messages that i'm just publishing right now so there's 21 and 22 23 and 24. let's jump back over 25. so here's all the new messages that i've been publishing so if you look at your database and i'm going to i just have my demo database here there are two tables that have actually got created by cap so there's one called cap published and cap received i didn't have to do anything with these these were just automatically created when i started running the app cap kind of dealt with it obviously there's configuration where i had to give it kind of the connection string to where this was all set up but it dealt with the actual table and you can see here uh for the published events these are the things that i've been publishing and it's marking if retries are occurring when it was added when it expires so it can delete it um because it's kind of like a temporary thing and then the status of it whether it worked or not but here's all the messages that i've actually been publishing so this is actually in my primary database along with my orders so there's one issue with the old box pattern that we need to address and we need to talk about is maybe you noticed it or maybe you thought about this already is that when you have this separate process or thread a part of the library that you're using which is fetching those records from the database and then it is publishing those to your actual message broker queue and then it has to go back and update the database for that record that says hey like this worked or it didn't work whatever the case may be is if it worked but it couldn't actually update the record in the database then that means that you're actually going to end up republishing that same event to the message queue again because it doesn't look like it actually was published so this is an actual situation where you're kind of potentially if there's failures with the publisher go going back to the database to say hey it worked you could be publishing the same event or message more than once i'll definitely handle dealing with duplicate messages in a separate video but if you enjoyed this video please give it a thumbs up if you have any thoughts or questions make sure to leave a comment and if you're into software architecture and design make sure to subscribe thanks you

Channel: CodeOpinion

Views: 18,851

Keywords: Outbox Pattern, software architecture, software design, cqrs, event sourcing, design patterns, software architect, .net, .net core, soa, microservices, message queues, kafka, event bus, event driven architecture, azure service bus, rabbitmq, distributed transactions, service bus, mass transit, nservicebus, message queue, messaging patterns, service oriented architecture, microservice architecture, enterprise service bus, outbox pattern microservices, Outbox pattern c#

Id: u8fOnxAxKHk

Length: 9min 9sec (549 seconds)

Published: Wed Nov 25 2020

Reddit Comments

[u/zeta_cartel_CFO]

Love your channel. Have learned a lot on things I couldn't quite get my head around through other sources. Like CQRS.

[u/mxjq2]

Thats really cool, thanks.

What do you think of accepting that you may publish events twice and making sure your services are written in a way to expect events being published at least once?

[u/bobbyQuick]

Watching you do that in a few lines of code made me cry. We use go at work...

I’ve been looking for this solution for a while, makes perfect sense and you should always be able to handle duplicate messages anyway!

[u/GenericDev]

Your channel is a goldmine! You have filled a few evenings of “watch latter” content! *subbed! Thanks :)

[u/CyAScott]

I used this pattern for interfacing with multiple systems in a way that made them seem like a transaction. I did it with NServiceBus sagas. My saga would keep track of which actions it already performed. If the handler encountered an exception while processing the request, a series of retries would happen in order to finish processing the request. Each retry would resume where the handler left off. Most of the time, a simple retry would fix the issue.

Like you said, it is possible to publish duplicate messages this way, but that's why we designed everything to be idempotent. The only issue we couldn't avoid was sending API calls to a 3rd party. That 3rd party API may not be idempotent which means we may accidently calling it multiple times. I remember this being an issue when a 3rd party introduced a breaking change in their response model. We encountered an error when deserializing the response, which meant we were calling their API multiple times because of our retry policy.

[u/[deleted]]

Get out of my head! :P

This issue was actually brought up at work, and something I pushed (unsuccessfully) to fix at my last job because we saw events that got dropped on the floor because of exceptions (my favorite was an issue where something wasn't SNS client wasn't registered as a singleton and something else wasn't awaited properly and the SNS client got disposed of before sending the message, whooooops).

I'll have to look at CAP and NServiceBus as possible solutions rather than using the of toy solution I wrote up (was fun to build though).

I'd be interested in seeing a commentary on the inbox pattern. For the unfamiliar, that's kind of the opposite where an inbound message is recorded in the database and a separate thread/thing handles processing it.