Building event-driven Java applications using Redis Streams by Mark Paluch

[Music] with the my session my name is mark palü I'm working with pivotal I'm spring data project Li there and I'm besides that I'm also project lead of the letters Redis problem so welcome to my session about Redis dreams with Jehovah and a spring datum so what is registries well rather streams is a fairly new feature that was introduced with Redis 5 and if we put that this way red streams are look like data structures and this isn't something which is totally you but we might know that from other messaging like systems like probably Kafka and that's where all this inspiration came from to build a very lightweight system a very lightweight data structure that mimics the patterns that we are already familiar with producing log events in append-only fashion and then consuming this this log to be able to read from arbitrary positions and this is a very powerful data structure as a turnaround we can use it for simple use cases like message passing to produce a message and then consume it on the other hand we can collect metrics and tracing information and logging information throw them in this local like and journal like data structure and then consume it later on with a different with a different processing mechanism like with a stream processing we can have something like an event or commit log that we capture incoming events store them in a red stream itself and then somebody else is going to pick up on this commit lock and processing events based on of that so that's basically what other streams is so Redis itself is well you could say it it is a key value store but it is much more than just their Redis is an in-memory data store which provides us with a couple of data structures and read streams is just one of those rather has this typical notion of a key values it has lists it has sets and sorted sets and so our adding streams was actually an easy thing to do because it's yet another thing that we can arm handle and if we imagine streams as a lock like data structure this would look like that way that we have something that encapsulates our registry this stream is stored under a key this is probably called events in our case and within this streams we have multiple messages messages are written sequentially they have an identifier this is this number over there and there you have a body and the body is a hash like structure but not entirely so you can have one too many of these boolean trees and the thing there is it isn't entire behaving like a hash but you can have multiple keys and named with the same name so that's probably but weird if we tend to think as the body is like something like a fundamental thing to registries is they are append-only so you cannot insert any message in between because you want to guarantee some particular total global ordering how events have arrived because you want to stream and read over these events and in that particular order and we have another concept that you might be already familiar with which is consumer groups I will tell about consumer groups a little bit later so how do I operate with that now the typical pattern how you deal in Redis with data structures is through commands and the same is true for Redis and streams itself so you have a handful of commands they can use to operate and interact with that data structure so that all that commands to enter integrate and interoperate with registries are prefixed with the X so if I want to add a message I call the X Air Command if I want to read from the stream well I call then the X read read command I want to know how many items how many look items I have in my stream I call the X length command to get a feeling for how many data is in there I can also introspect into that stream with X range to say I'm just interested in checking what messages are in a particular ID range or I wanted to probably get rid of messages so I can trim that content with the extreme command and say please retain me only the last hundred or thousand messages and then res world truncate your stream and trim it to to the appropriate side let's take a look on ready screams on the CLI first and hopefully Democrats and internet gods are with me because I turned out that the internet connection here the box is sort of shaky cool so I have this Redis CLI here and let's create a stream stream entry so I'm calling the eggs add command giving them the name let's call it my stream now don't pay attention too much to the asterisk I will come back to that in a minute and then let's give it a body but just key value and then here we go so what happened here is that I have added a message to the stream named my stream with a body key and value and in return I get this ID I can't do this this multiple times and every time I add something to the stream this ID keeps incrementing and the idea behind this ID and with that with this - here is to avoid collisions in a second or millisecond range so if two producers can produce the sale in the same millisecond message then this last part will be has been incremented all right now want to iterate or introspect my stream that the x-range command tell it my stream as keen I'm from minus to plus which means from the beginning to the end and now I see I have a couple of these messages I can also add another like a key value and introspect then my my stream and there see I have this two keys named key and this is I think some one of the inconsistencies that we have real with already screams because in every single programming language that uses Maps I would use either the first pair key value pair or the second one so this is something that could be perhaps optimizer all the time right second thing is well I mentioned don't pay attention to the earth Asterix no we get to that point so Redis itself is able to replicate messages and to replicate its content so so you you have a primary entropy car concept or with in Redis and the way Redis does this is by just streaming the commands to the remote side and when we create a message on on a on a peer or we get back this ID and we want to make sure that during replication we propagate that ID also to to the remote peer so that's the use case for either saying asterisk which means auto-generate me please an ID or I could also pass in some arbitrary ID like take this one okay and now I obviously did something wrong well actually that's that's a use case for our for putting Asterix there and using X length let's see how long our stream is my streams then we would have six items and so that's the registry Li part to the very basics commands to streams now this is isn't really necessarily the way how I would consume a ready stream but rather what I would be more interested in is in consuming messages in a linear fashion fashion so as soon as messages come in I want to read them and consider and for this use case we have this X read read command where I say okay X read from streams my stream and start with 0 0 position and then I get a lot of these messages streamed in and this is just one one way to invoke it the other way is well once I have received and process all these messages I probably want to go on with the next messages so what I would do is take the last idea that I have read use that as an input command and then because nobody else is producing messages currently I don't get any message or right now to not pull over and over and stress my Redis server with appalling commands I could also say ok let's block this read for like five seconds and when within five seconds somebody is going to produce a message then I would be able to receive that and since nobody is doing that well I'm getting a sort of some sort of timeout now let's activate yet another registration this one prepare prepare this adding two to the stream which is my stream go into docking read mode and now as soon I as I publish that method this connection gets unblocked and I'm receiving my my message again we'll just fine thing to do but I have to to choose very invasive my battle and my powers in how I'm going to consume read streams and this is basically do I want to Paul over and over do I want to block that connection or what is my real reading strategy and we will also see that the same pattern within a Java code to Mirage so let's switch over to some Java code if anybody cannot read that because of font size give me a hint otherwise I will continue so I have here this empty main program let me spin up a Redis client and you know because I'm maintainer of letters so that makes a lot of sense to use also my client now what I've done is I've created that letters client instance connecting to Redis and I'm using the synchronous API to interact with radius and then I want to ya want to produce messages and if I would run that code um right now with without anything else I would just write just a single message but you know because streaming is about continued DT let's wrap that in a while true loop and execute that crowd so what we are doing here is basically creating a body which carries the actual time the current time and we are adding that body to the my stream data structure let's run that and then we will see and observe some activity happening here at least on the console so we are every second we are appending a message to the stream now if we start reading then we suddenly get a lot of more messages so something is happening here at that end which is cool so producing stream messages from Java is that simple just called an command give it a name give it a body and there you go now if we start thinking about how to consume that from a Java perspective let's get rid of that we again need Redis connection and we need a little bit of code to consume that and you remember perhaps you need to remember the ID of the last message you have read so and this is sort of cumbersome but let's let's go for the time being with with that idea so we again need some loop to keep that program alive we need to remember which message we we've read lost and then with these information we go to the command interface called the read Mesmer method we are blocking for one second to not stress our service too much and what this code then is doing is it is printing all the messages onto the console and once it has read a message it is remembering the last message ID and then reusing that to read from that offset again now let's run that and then we see a lot of things are currently happening that's clear the console yeah now we get these messages of streamed into our program basically sort of simple but you know what about this idea here what about this last message this is typically something this is a state we have to deal with that state and we don't want to purse it this that into some sort of database because you know distributed transactions are more evil than good and they have traditionally caused a lot of issues with that and this is the point where exactly consumer groups come into play so if we go back to our mental model of ready streams which is basically locked articulate data structures we have these events which are part of the stream they are stored within this man particular key and now we get to the concept of consumer groups which is basically yet another maintenance data structure within the stream itself that is able to track the progress and to track the actual our ID that I have consumed so the concept there is that I create a couple of or one or multiple consumer groups and each consumer group is identified by a by a name within a consumer group I can have multiple consumers and each consumer has has a also name and it is associated with the last seen message ID so with consumer groups we get a tool to persist our message ID to track our latest or most recent message ID in a way which isn't invasive to our application but rather we can stop reading or our application is able to crash and we are able to resume from that point on where we left the last time and the idea behind consumer groups is well how would I use that perhaps a consumer group would be my order application my logging application my whatever you you have application and the consumer idea itself would be perhaps the name of the service instance so if you run multiple service instances that would be a good thing to to put into that it's not a good place however to use random user IDs or the like because you won't be able to resume from from that point on so perhaps host names these kind of things would be a good consumer groups you can also interact with these consumer groups in the way that you have this X group command which allows you to create consumer groups to delete consumer groups to update the state the last on the most recent ID and you get yet another feature with consumer groups which is acknowledgment so thing is when you consume a message through the x-ray read command this automatically means to Redis that you have seen and successfully processed that message but what if you have to work a couple of steps to process that message in that case something can go wrong and yet Redis has noted that you have successfully process the message consumer groups help us with that because with the X read group read group command you can pass in a flag to not auto acknowledge a message but rather you get out of that message this message gets reserved for you as a consumer and consumer group and then eventually you call expec to acknowledge the message processing and then the signal for Redis to increment the last message counter and to signal that you have processed that message successfully cool so this is the foundation of Right Scoop so what can we do that I think we can do we can entertain the audience and that's I guess now where I need your help I think most of you have either notebook or a mobile phone or iPad so get that prepared and connected with the Internet because I mean we all need a little bit of hope for you because we are doing real-time Map Reduce stream oriented may produce some 20 rod so basically what I'm trying to do is I've prepared a website I'm not the best one on your eggs so bear with me about that and I thought let's make an online and live poll of which is your most favorite feature in Redis so there's a website for that I will show you show you the link in a minute you will be able to select one of these submitting about and this is going getting on Britain to Redis itself and on my machine I have a stream processor more I need to activate it yet which will be then consuming that stream and evaluating how many volts we have and in the end we will know which is the most popular feature or which feature you like the most so how do you get there it's a bit leading which is reddish Redis - screams - perhaps if the QR code works then right otherwise I guess you need to type it in and now let me go to my java program launch it and this is to be honest spring boot thing and we are from the spring site very much into application efficiency so you see things are currently happening in there and how do we consume that input in in Redis in a register fashion well we have this thing called reactive programming which treats data sources as a stream of data so instead of having some Lube that closed my code we are able to consume that read is stream a stream know that that's again so with a stream of a stream we are able to flat map things there are which means we are starting a synchronous sub processes this means every time you submit a vote we are processing that vote and adding that into a sorted set to increment other poll stats which means that I'm able to reduce that votes into a sorted set which gives me then a pretty good impression well how how you have ordered now let me do my vote as well where are we you know you see if things are happening there that's that cost me my vote yeah ready streams rock whoo and as you see lots of messages are coming in and streaming in which is good widget because Democrats are on our side let's see where this is stored ok this starts in the keep all stats which is of type set set which it means sorted set and I can read that with set range 0 to minus 1 well with scores and this means to to report how many hours we have so we have like 122 volts for Lobot 103 for swims and that pup is May cool so thanks for your participation on on that so I think this should give you a pretty good overview what you can do in a very easy fashion with ready streams and one thing and one problem I have with registrants itself is well you have always this loop the spalling thing so you either pull it yourself in your code or you let somebody else do that and if you are into Java and you are pulling in a synchronous manner then this means you keep that connection busy and you keep a threat busy and this isn't the most efficient way how to do deal with that kind and if you pull on a single stream or maybe a to stream on two streams you keep to resources busy and you if you pull on a thousand screams you keep a thousand resources and thousands threats busy see this doesn't scale well and this is an issue I currently have with streams and so the other thing is Java isn't the only language out there well maybe spring data has solved that issue for the Java community or the at least the spring community but there is hours there go there is no js' there is Python Ruby are all these fancy reference your languages that you could imagine which have integrations for Redis and if you're part of that community if you want to give back to the community if you want to contribute then I really encourage you to pick up on these ends to make lives for your peers for your users easier by providing a similar abstraction that takes away this burden from regular application developers because you know regular application developers want to focus on their business problems on their business domains they don't want to keep threats busy they don't want to do pointless loops and these kind of things so my message to you today as a takeaway so this brings me pretty close to the end of my presentation all that code that I used today during my presentation is available on github it's all open source the slides are as well available on speaker deck so you can grab them and go over those afterwards you can find me on Twitter and github under my handle and this my friends is it thanks for attending and if you have questions we have still a little bit of time to answer these yes okay let me rephrase that okay we have that possibility to limit the number of messages in the stream but what about retention time this is a good question registers not have that feature yet Redis has a timer itself built in and I would propose go to the Redis issue tracker file you back a report or a feature request for that because I think that would be something that would make sense yes okay so the question was is there a process to keep register synchronous so you can use streams within the class or I'm not exactly sure whether this is the answer that fits your question so you can use read streams within the class itself and the way Howard's cluster works is it charts by key so your entire stream lives on a single short note and due to this replication mechanism Redis cluster replicates the same message IDs to to its replicas does this help okay okay so so radius itself keeps its synchronization on an asynchronous way so you it is eventually because instant perhaps okay cool one last question pardon okay why would I use Redis instead of Kafka so I think there are multiple ways how to answer that question I think one of the most common ways is it is very easy to set up a Redis binary to read the server you just spin it up and it's there so it's very low maintenance so if you run Redis which is I guess true for a lot of shops and a lot of setups then you just use yet another feature from Redis while as setting up to half car is isn't the most trivial task so so that's my take a take of that in any case registries is quite limited when you compare it to - Kafka because in Kafka you have this exact ones and delivery or at least once or at most once at the all these nodes these modes aren't necessarily reflected in Redis streams itself so if you grow to a point where you need these features then Redis is probably not the right tool but if you are into just streaming and logging things with our delivery guarantees then probably Redis would be a good good choice and that being said thanks everybody Android that works [Applause] [Music]

Channel: Devoxx

Views: 2,956

Keywords: DevoxxUK, DevoxxUK2019

Id: NUwrSrYxFGg

Length: 27min 55sec (1675 seconds)

Published: Thu May 16 2019