Redis as a Primary Database

presentation please okay that's me all right Ready's as a primary database well as you all know ready started as some sort of caching system and gradually evolved to become a database we attract slabs make a lot of effort to make readies a primary database for as many used cases as we can and today I would like to share we would like to share with you some of the innovations and demonstrations associated with Redis as a primary database and I'll start with the functionality of Redis as a multi model database well modern application use what we call polygon persistent architecture by which multiple micro-services communicate with multi one or more data services databases and then those databases communicate among themselves and send data to one another the limitation of this approach are first of all there are too many network hubs which may which may make the application slower secondly you have too many data services too many connectors too many glue components and it tends to get complex this set up it tends to tends to get complex and also expensive because on the cloud you pay separately for each of those micro services and for each of those data services another modern approach is to use what we call a multi model database architecture by which the micro services communicate with a single database which has all those separate functionalities of multiple type of databases and this makes things much simpler of course in the case of Redis we implemented the multi-model functionality with a novel concept which is called Redis modules which was mentioned here before basically we have a core part of the database and then an API which enables anyone to add any C C++ Python go library to the database and make it an integral part of the database now the nice thing about this approach is first of all it's modular so those those those different type of database functionalities are not tightly integrated as you find with some of the other multi model databases and secondly what you have here with red is each of those modules have the real functionality of a graph JSON search etc it's not just an API ok and we have you know with the module system we have people from the community built already many different modules we have some enterprise customers that build their own proprietary modules even without sharing this with us and then we as a company build modules which are intended to expand the use case coverage of Redis and make it multi model we would like to show you today how this works in reality and for that we have built an application which takes the Twitter firehose and does the following first of all word count on the Twitter firehose indexing and then searching and then building a social graph out of the data let's see first how this could be implemented on the cloud on AWS without Redis ok so basically you take the data from the Twitter firehose feed it in two kinases which is the streaming engine of AWS and then you need to build a lambda function which will take the data from Genesis and move it to ElastiCache so this the red is shorted set can do word count for us then you will have to build yet another lambda function to take the data from ElastiCache and move it to dynamo DB for persistent purposes in parallel you need to build another lambda function to take the data from Genesis and to move it to cloudsearch for indexing and all the searching functionality and another lambda function to take the data from cloudsearch to elastic as to elastic cache for instant response and in parallel you need to build another lambda function to take the data from Genesis to move it to Neptune which is the graph data service of AWS and from there to build another lambda function to take the data from Neptune and move it to ElastiCache for instant response and on top of all that you need to build connectors from your application servers to each of those data services does this remind you of anything so you know in order to do that we used five data services six lambda functions 5 application connectors and we ended up with seven Network hops okay did I mention slow complex and expensive let's see now how to do the same exactly the same thing with Redis so for that we would use Redis integrated with two modules the first one is ready search and various graph and basically what we do is we will take the three firehose data feed it into the stream functionality of Redis and then use a sorted set for doing word count use ready search to do the indexing and searching functionality and raddest graph for building the the social graph very simple to do will show you that we end up in with one data service zero lambda functions one application connector and one network hope I think that it's it's no need no words are needed this is this is so much simpler than to do it with you know with with polygon persistent architecture we would like to demo it now and for that I would like to call Kyle Davis Kyle hey please come on ok all right let's see if everything's still up so what I'm gonna do today is take the Twitter firehose as we said and I wanna use it read a stream to pull it in and start using that to turn into a read a stream so as we can see here I'm doing that I just want to check on something momentarily yep we're getting tweets ok so then what I'm gonna do is I want to Det word counting thing right I want to create a word frequency chart so I'm gonna do that and let's take a look how that looks so we're live generating this particular word cloud and frequency chart as we go along by just using sorted sets and some interesting stuff like that you might guess what keyword I'm using I make no adherence to what the content is this is the live internet so there's that then we're gonna say ok we look over here this is some really interesting stuff right you can say ok what's going on with this so let's take a look at some something here you know think I just solve that keyword right so I'm gonna go over here I'm gonna look in search so let's go to the search engine now turned that on now we're indexing live as we go along so let's take a look at search and how that looks so as I type I'm searching through this stream of data we can see see everything here this is using Redis search is a real-time search engine that we've built I can go in and get let's see here let's do my crawling right that's what I was looking for thank you our oh man okay Baker Monica okay I have live internet guys screen name so I'm gonna use it as a secondary index here and we can see all the things that this person has tweeted which is really interesting right but you know we can also see like where they've mentioned as well so we can see and mentions and I can't type today we can do that so we can see everybody who's mentioned the ACLU for example that's really interesting right but what if we wanted to see for example how these all related right that would be a graph problem so I can go ahead and go over here will turn graph on and then we can see how everybody is connected so I'm gonna make this a little bit larger here and let's take a look at the social graph of what we've been ingesting so then we can see the social graph and let's take a look a little closer at this zoom out a little bit more and this is all built using streams and this is Redis graph which I'm going through and seeing the relationships between people who have mentioned things and we can see some interesting relations let's take a look here let's look at the hill which is and we'll take a look here recenter that we can see whose set things here and we could take a look at one of these people and see who they've interacted with as well probably not a lot of people that's some things work on Twitter and we can see they just interacted with this so as we look at this we can go back through and see everything here so in this presentation you've seen Redis as a social graph using the reds graph module as a search engine which is not the screen but this screen and as well as using sorted sets to create these word frequencies so we're using it as a truly multi model database

Channel: Redis

Views: 7,894

Keywords: redisconf18, redis, redis labs, redis enterprise, redisgraph, primary database, ofer bengal, multi-model implementation, data, databases, NoSQL, opensource, tech, technology

Id: SGEG5q4LT9s

Length: 11min 26sec (686 seconds)

Published: Wed May 23 2018