Building Lightweight Microservices Using Redis

welcome to building lightweight microservices with lettuce thanks for attending this talk and thanks to Redis Labs for having me I know it's late in the day and I promise you a lively presentation that you started a bit about me my name is Carlos Cheston ya know I hold of 2005 guinness world record and distributed computation blah blah blah blah I won't bore you with the details you can find out more about that on Google I'm also chief architect that's of flywheel sports I've been a Redis user since about 2011 you can find me on Twitter and github I see just and on c jme before we dive in I'd like to quickly clarify what I mean by lightweight micro-services light on external complexity and dependencies meaning microservices which limit the number of external software and infrastructure dependencies not light as unlimited so micro services which are not limited and what they can do to disclaimers are in order the first is that the approach I'll share with you in this presentation may not be a good fit for your projects you may need heavier micro services and that's okay the second disclaimer is that the first half of this presentation may seem a little bit fluffy but trust me you're in the right place and I promise you will get increasingly technical don't say I didn't warn you so as I said I'm with flywheel sports flurry was founded in 2010 in New York City as an indoor stadium cycling studio our fitness programs feature high-energy workouts that will challenge even seasoned athletes a cool thing that we do is add sensors and microcontrollers to our bikes in order to track performance metrics so we've been doing IOT for the past eight years we have roughly 1200 employees across 42 studios in the United States three of those studios are right here in the Bay Area one on Market Street here in San Francisco one in Sunnyvale and another in Walnut Creek you can try a free class on us using the offer code for much of what I'll share with you today has been proven at flywheel sports and a group of other companies so I'll use flywheel is a case study during this presentation last summer we prepared to launch a fly anywhere platform which allows our customers to ride at home using our nationwide live video streaming service how cool is that now while we're not the first movies in this space we are aggressively innovating here's a quick short pre-launch clip that we posted on social media [Music] [Music] [Music] we launched our fly anyway our client tells how you ride deliver your classes based on your goals while our tour and so you always know where you stand among our community of over half million never coasters anyway because limits were made to be broken we're proud of our bike tech and nationwide launched a fly anywhere and we're pleased that we did this while embracing micro-services and Redis although things didn't start out that way just over two years ago we started thinking about embracing microservices the catalyst for this was our CTO Mike Rolando who was recruited from Nike where they built micro services before joining flywheel I learned about their existing monolith now a problem with all monoliths is that they have a tendency to continue to grow over time and that's exactly what happened that flywheel during a five year period working with a monolith can be you know challenging as code becomes highly interdependent over time this can lead to bugs and in some cases if you're not careful a combination of bugs and interrelated code can lead to catastrophic failure as the entire monolith goes up in flames true story but it's also a worse case it's worth noting that the construction of monoliths are often unintentional no one says hey let's see how big we can make this thing the truth is the increase in size is often the result of convenience an alternative approach is to build modular and distributed systems the microservice architecture is one such approach which embraces both focus modularity and distributed computing the end result is the collection of loosely coupled systems the resulting modularity is great however the distributed nature introduces its own set of challenges namely that communication is also distributed and so network failures and system availability become important concerns still we were clear we wanted to choose micro services over monolith after all how hard could it be I'll let you soak in this slide you pat yourself on the back if you're familiar with each of these terms if you're like the rest of us some of those terms are still in your checklist now you don't need to know all of this in order to build micro services but they are important tools and concepts so there was a lot to consider and unfortunately we were a small team with limited resources that's three strikes if your accounting experts will tell you that small teams should not embrace micro-services one reason is that the microservices approach adds to their already busy workload that developers have there are you know lots to consider while embracing microservices such as you know how services will work together be monitored in nd blood and how their data sources will be organized so a core challenge is that there's much to learn which amounts to a lot of study time for an already busy team so those were certainly concerns for us we were also well aware of the great solutions that larger companies developed an open-sourced however even those gifts from the gods come with steep learning curves so a reality check was in order when on Netflix and chances are you're not either naturally if you are Netflix then you have Netflix size problems which require extensive engineering by large teams given our size development budget and real business priorities we needed to right-size a solution we needed something small and robust to quickly get us started while affording us a runway so that we could later embrace other solutions as needed in his book The seven Habits of Highly successful people self-help author Stephen Covey wrote in habit number to begin with the end in mind that is begin by understanding the outcome year after then looked then work towards that outcome so we started by assembling a checklist of features we thought we needed if you already work with micro-services then this checklist is probably quite familiar one of our most important goals was that an ideal solution had to be simple to use by less experienced developers ideally implemented as a single package import so I've been working with Redis for some time along the way as I considered our checklist I decided to take a closer look at Redis I started seeing Redis as a sort of distributed computing glue it looked like Redis could be used to bind and unify micro-services each micro service binds to a shared Redis cluster presence is maintained with key and exploration inter-process communication via pub/sub routes placed in set stores but was it possible micro service features using a single infrastructure dependency we thought so so we built an internal library called Hydra the name Hydra seemed ideal because a service type often has multiple instances working together as a larger beast on the right side is the list of features we were able to support by the way you'll be hearing the name Hydra a lot in this presentation because I'll use Hydra as a practical example of how Redis can be leveraged to build micro services let's take a closer look here we see three micro services each with a Hydra module which connects to Redis in this model most services don't commute communicate with Redis directly rather the underlying Hydra module proxies to Redis another point about this diagram is that Hydra is just another imported module just like the one shown in green Hydra is only shown in blue at the bottom to illustrate his presence and connection to Redis here's a scale down view of Hydra as a JavaScript class there are actually 20 GB 36 member functions in total but this snapshot provides a sense of the ease of our abstractions so Hydra with a single module import and fairly simple class methods allowed us to choose ooh away at our monolith and start building our fly anywhere platform as a collection of services powered by Redis fast forward nine months fly anywhere was already well underway and Hydra was working really well for us and we felt it would be useful to others so on November 7th 2016 at the Smithsonian Museum of the American Indian in downtown Manhattan we open sourced Hydra on stage at the empire node conference i demos hydra running on a cluster of raspberry PI's my point in building the cluster was to reinforce the notion of a lightweight micro service library in fact light enough to run on five dollar raspberry pi zeroes here's a closer view of the cluster it consists of eight computers seven of which are raspberry PI's and one of which is an Arduino based microcontroller each of the PI's are running Hydra enabled micro services using node 6.5 in total the cluster contains 16 CPU cores in order to further add context to this presentation I'll briefly show you hydro in action then we'll look into the hood to see how hydro leverages Redis to form a lightweight micro-services library will drop down to the command line and launch a docker swarm cluster consisting of a dozen or so micro services some of the core services include neo4j MongoDB and Redis and the rest are hydro enabled micro services we'll give that a bit one of the services though will be interested in as and see in action is the hydro router which is that dynamic service aware router next we'll launch his dashboard via the provided URL here we see the hydro router dashboard if we scroll the page we can see all of the services we launched so we see an image processing service a notification service and odor service these are not service is used by flywheel these are services used by a buddy of mine who's doing work for one of the largest printing companies on the East Coast so I used the services for this demo so if you look closely you can see that the hydro router instanceid and a port or the same as shown at the top there if you look above the the input line you see that it's Hydra out at one point six point nine experimental but you see that the the ID and the the instance ID match what's shown there we can ask the hydro router for its version by typing V one router version so hydro router accepts restful calls that's a version we can see itself information via v1 router health so some pretty useful information we'll get back to this and we can also see a list of nodes that are running so you saw the graphical view of these nodes this is the actual JSON data set that comes back from from them we can even see the API routes for all the registered services so that's another thing service is a registering route you can actually produce them one of the cool things we can do is to speak to other services in the cluster true hydro router here will ask the asset service to return its health information we can do the same for the auth service so will seeing that hydro router itself a hydro service is able to monitor the presence of services running in the cluster let's kick this up a notch on the center on the top center we see that we have only one instance of the off service running let's launch more instances first let's check to see how many instances are running we see that there's one of one running shown there on the right next we'll use the doctor service scale command to launch two more instances for a total of three switching back to the dashboard we see that we now have three instances of the off service winning zooming in we can see that each instance has a unique instance ID now we can call the help endpoint on the off service if we refresh the page we'll see that we get back different results for each of the different instances our requests true hydro router are being load balanced across available off instances now let's drop two of our auth instances to see what happens so we're going to scale it back to one off instance and we'll just use a darker scale command here as well they're setting it to one [Music] you know okay give it a second we see that two of our instances are now in red indicating that they've lost presence and we're left with only one instance with an ID starting with 82 and ending in 35 let's take a closer look let's try our health check calls again so as we refresh the browser we see that we get back a response from instance ending in 82 should be starting with a 82 and ending 135 refreshing the browser shows us that that's the only instance that's responding because the other two are unavailable let's tell how's your router to clear the debt instances that are shown in red we can do that using the v1 router clear end point so that was presence and load balancing in action now let's try messaging for this demo we'll use the HMR service which simply relays messages that every see is back to the sender it's a good testing um app for messaging give that a second so we'll open a WebSocket client that's just a phone plug-in and connect to Hydra so we do that by just typing WS using the WebSocket protocol in this case I say localhost but actually it's going to the darker cluster so what comes back is connection messages a message from hydro router so we say it's type connection we also see that it was sent to the client there on line 2 and it is given a client an ID also it's telling us that it's from the hydro router at 0 8 ending in BA I believe so now will send a message to the HM our service we structure the message with the - we tell it hey it's coming from the client the mid-20 and we're basically sending a test message just says hello and we just click on the send button and if we scroll back to the bottom we will see that we did receive a response of two hydro router but it came from the HMRC and it gives us the instance ID of that service we also see sort of a text description of what just happened showing the instance ID that came back the last demo I'd like to share with you is the use of TCP dump to see the underlying Rattus packets moving between Michael services I wish you a dump command password required there we go here are the packets moving between micro services and Redis so if we stop the trace we see the actual details we can see various bits of TCP data such as the sequence and length of the packet and we can see Redis commands such as set X and H set a key takeaway in these demos is that you know what you've seen is powered by Redis at a much lower level the last thing we'll do is tear down the cluster and continue the presentation as you can see Hydra is doing quite a bit with the help of Redis let's deep dive into how Redis can be leveraged to enable each of these features keep in mind that you don't need Hydra each of these features is made possible using Redis and you can certainly do this in your own apps what Hydra does do is automate and abstract the underlying details that's something you should strive for if you build or use a similar library another key point and what I'll show you is that some of these features are only made possible when combined for example request and message routing depend on presence health service discovery and load balancing as you know each of these features can be addressed using various infrastructure tools however one key and that advantage of and what I'll show you is that these features can also be implemented just using Redis and your favorite programming language as we take a closer look keep in mind that there is a meta context that binds higher level abstractions and there is a wonderful word which describes the end results synergy when when the whole is greater than the sum of his ports we believe that this is especially true with Redis early on it became clear that some thought needed to go into how to organize the Redis key space Hydra uses keys which consist of names with two to four segment labels separated by a colon character the segment labels a name prefix service name instance ID and type the prefix segment allows for filtering Hydra versus non Hydra keys if you make heavy use of relased than being able to filter for specific keys is vital the service name segment aids in filtering keys for a particular service type examples would be authorization user or an image processing service the instance ID segment allows filtering keys for a unique service instance when running micro services you typically want multiple instances of a service type running each service instance is assigned a unique ID and being able to differentiate between them is useful lastly there's a type segment which is used to classify the purpose of a key not all of the segment's are present in each key for example service name and instance ID are not required in some keys here's an example of the key for the user service we see the prefix which is Hydra service followed by the service name in this case user service next we see the unique instance ID and lastly we see that the type of this key is presence so we say that presence information is stored at this key address we can enter the Redis see a line and type Redis commands to view various keys we'll see examples of this throughout the remainder of this presentation to recap our use of keys in Redis is organized by segments which makes them easier query furthermore a consistent organization makes them easier to extend and maintain as we continue we'll see the role that Keys play in the organization of each micro service feature let's begin by examining presence in the world of micro-services the ability to discover services than to know whether a service is healthy and can be routed to is of paramount importance those features depend on knowing that a particular service instance is actually present and available for use this is also required for features like service discovery routing and load balancing once the second Hydra updates the TTL or time-to-live of its service key failure to do so within a 3 second period will result in the key expiring and the host app being perceived as unavailable we can see here that the Redis commands are just get instead X which set the key and expiration we can query presence keys using the keys command with a pattern match note that there are three keys present this tells us that there are three instances of the off service running or asset service in this case if we attempt to retrieve the contents of one of those keys we see that it contains just the instance ID using the TTL key command against the key shows us that this particular key has two remaining seconds before it expires so to recap presidence can be managed using keys which ought to expire updating the key is done automatically by Hydra on behalf of the host service meaning it's not something that developer does failure to update the key within three seconds results in the service being perceived as unavailable that probably means the service isn't healthy which brings us to our next topic being able to monitor the health of a micro service is another important feature Hydra gathers and writes health information snapshots every five seconds you can you can quickly check a snapshot to view in individual services health details and the snapshots can be used by monitoring tools such as the Hydra router dashboard we saw earlier so here's what the health key looks like notice that the only new bid is the type segment identifying the key as being about health when we view the contents of the key we see that it contains signified JSON text in this case it's for the project service on string of finding the JSON makes it easier to see what's stored it contains lots of useful information so health information can be stored per service instance it's managed using a service key or a string key rather that contains stringify json text and that information can be used by monitoring apps next let's consider service discovery which is another must-have feature for any microt service architecture the ability to discover the IP address and port of a service by name greatly simplifies communication other bonus points include not having to manage DNS entries or create fixed routing rules service discovery information is stored in a Redis hash with a type of nodes the use of the hash enables blazing fast look ups we use the raddest HKH status get all commands to work with the notes hash the following Redis operations can be used to implement service discovery the first is the lookup for a particular service type the second is a lookup for available instances and the third lookup allows Hydra to retrieve information about a specific service instance we can see useful information such as the version of the service the instance ID IP address port and finally the host name in this example the host name also happens to be the the dock of container ID ID we can retrieve information about all available instances using the Redis H can all command this is how hydro router retrieves a list of services to show on its dashboard so no magic so let's recap Hydra queries using the service name key segment in order to discover various bits about a service service details can be managed using a Redis hash which offers blazing fast look ups next let's consider routes for routing Welding both HTTP messages and web socket or pub/sub require that routes be validated micro-services can publish their routes to read as we saw that earlier as an example Hydra uses the published routes to implement dynamic service aware routing each service publishes its key routes to a key of type service routes we see here that the key shown is for the asset services routes service routes can be stored in a set structure a good choice because you don't want duplicate route entries the s and s member commands are used now as an aside Redis is rich collection of data structures is one of the reasons that what I'm sharing with you is even possible returning back to out we can pull a list of routes using a key pattern here we see the routes for a number of services we can use the s members command to view the contents of a specific route set by the way the get post input bits represent HTTP rest endpoints for other messaging transports the use of bracket methods can be omitted so let's recap each service publishes its routes to a Redis set accessing an individual reveals a collection of route entries for that service voucher stored in Redis using a set data structure which avoids duplicate routes the published routes can be used to implement service aware or dynamic routing next let's consider load balancing as your application grows you'll need to load balanced requests among available service instances this is accomplished with Redis using the service presence and routing features we've seen at an application level using Hydra this is as simple as using the make API request if it's restful or the send message call if it's a pub sub or just JSON message load balancing takes place inside those calls as Hydra uses routes and presents information to choose among available target instances a nice benefit is that during routing if a request fails for a particular instance Hydra is able to reach by other available instances before returning an HTTP 503 server unavailable error as you can see here load balancing relies on other features such as presence service discovery and routes when presence is queried individual service details can be retrieved here we see that for the asset service we can obtain his IP address import to recap load balancing requests among services can be accomplished using presence service discovery and routing features we've already seen where the strings hashes set make this possible again the whole is greater than the sum of his parts distributed services are forced to communicate with one another over an underlying network because they're distributed HTTP rest calls are probably most common but socket messaging can be far more efficient messaging in Hydra is accomplished using Redis pub/sub channels and Redis implements pub/sub over socket connections here's an example key Hydra uses the subscribe unsubscribe and publish commands along the way as an aside the Hydra water is able to accept messages over HTTP and WebSockets and convert them to pub sub messages we saw that earlier to understand how this works consider two services the asset service and the project service each service creates two keys one using its service name and another using its service name and instance ID each service listens to both channels in most cases you don't care which instance of a service handles your request in those cases the channel without a specific instance ID is used now when you need to send a message to a specific instance the channel with the instance ID can be used it's important to note that Hydra converts requests to a service name to one with a specific instance ID when it load balances that ensures that only one instance handles a given message or requests we can see a list of channel keys using the Redis pub sub channels command notice that we have four keys here the first is the name of the asset service shared by all instances of the asset service next we see three additional keys with unique instance IDs one for each of the three service instances continuing with our focus on messaging in order to ensure interoperability between micro services it's essential to standardize on a shared format of communication the universal message format is a documented JSON based format which includes support for messaging routing and queuing these messages are stored in Redis as stringify JSON text here's an example of a umf message the - from embody fields are required and services are free to customize in the body portion of the object and we saw this earlier when we send messages WebSocket messages let's see how this is used in practice on the Left the client service sends a message to the project service note that this only requires aumf creation call and a send message call shown here in yellow on the right side the project service listens for messages and processes them as necessary that's accomplished using event and event message listener note that Hydra abstracts away the service discovery load-balancing routing and pub sub specifics sending and receiving messages only involves tree member functions so it's worth pausing here for a brief moment take a few seconds to consider what this example would look like using your favorite stack let's take a closer look send message works by parsing the - field to determine where the destination of the of the service with the service theme in hand the next step is to check for available instances with the target instance in hand the messages then stringify and sent via the Redis published command again we can list all the pub sub channels in Redis messages can be sent via these channels and retrieved by listeners so with a bit of programming code were able to use Redis to route messages using a well defined collection of channels in summary it's worth noting that messaging is eventually necessary because services are physically distributed Redis enables messaging using its pub/sub features standardizing communication enables interoperability between services we also saw how easy communication can be at an application level when we abstract away the underlying service discovery load-balancing routing and pub sub specifics next job and message queues are yet another important part of any non-trivial application Hydra uses Redis to maintain dynamic accused for each service type service instances can then be can then read their cues and process them the contents of a cute message is a umf message which allows for the same format used for messaging again interoperability is king Hydra automatically creates three queues per service type a receive queue and then process queue and an incomplete queue because these are lists and using Redis now we can use Yelp push or push or pop L push and LM commands here's a diagram showing the message flow between queues the movement of items between queues is an atomic operation in Redis so it's safe regardless of how many micro services you have on the Left queuing a message is as simple as creating a um F message and calling Q message to send it the code on the lower right shows a message processing the image is rather the image processing service D queuing the message just by calling get Q message and later marking it with mark Q message once it's completed with the message I mean how easy is that so to recap sometimes it isn't feasible to expect an immediate response in those cases we need to queue work for later the reddest list data structure can be used as a message queue commands like help push and/or papel push with atomic operations make this feasible here again we saw how easy basic queuing can be using higher level abstractions distributed logging is another vital feature of any micro services architecture however if you know Redis you might be appalled at the thought of using it as a distributed logger and you'd probably be rightfully concerned however you could use it as a flight recorder where you only store the most serious errors and you limit the number of entries using L push and L trim then at least you'd have a quick way of checking what might have gone wrong with your micro service here's what the key looks like notice that the the keys of type half log here we see the health log key is actually a list data structure so we can use the Redis tell range command to view the flight recorder log for the image processing service recapping having dozens of having logs across dozens or worse hundreds of machines isn't feasible with micro services distributed logging is definitely the way to go using Redis you can at least build a lightweight logger to use as a flight recorder using register list data structure and the handy out push and Outram commands make this possible now lastly let's consider configuration management managing the configuration files for distributed micro services can be challenging however you can even use Redis to store config files for your services we did this and it seemed like a good idea at the time however we're starting to move away from it as the core disadvantage is that storing configs in Redis makes better stateful and that's less than ideal but it is possible so I'd like to share this with you let's see how this works so there's a keys type which is the hash the hash has a key consisting of a service version and the value set to the configuration data for that version pretty straightforward there's a sample config in our case we use a command line tool called Hydra CLI which allowed us to push config files for a specific service version um all this really did create a hash entry with a key consisting of the service name and version and the file as a string of five JSON keep in mind that you can also use a shell script to drive the Redis CLI so using H key role we can see the config version for all the the asset service we can also pull specific version using the H key command with the version of the config so let's recap we solve Redis can be used to store application configuration files the Redis hash data structure makes that possible each config is indexed using the the service version label and the contents just contains JSON text well we've covered a lot but in summary what I've shared here is an approach that is that very heavily leverages Redis using javascript and nodejs however there's nothing preventing anyone from doing the same with other languages and because Hydra is open sourced it can serve as a reference platform for such efforts I'm pleased to announce we have a Hydra inspired go version and development which we hope to open-source soon now that's how Hydra leverages Redis and how we were able to both build and launch or fly anywhere platform I'd like to show you how it all turned out here's a photo of our Bluetooth enabled bike and metrics display so a key call out here is that our micro services are dealing with real-time communication our bikes that make real-time data which devices packaged into messages and transport by a WebSockets and Hydra water involves them to their target micro services we broadcast live from our production studio in New York City the studio utilizes state-of-the-art forecasting equipment similar to what you'd find in a new station however perhaps unlike other Studios ours runs a collection of micro services which support the automation of each live broadcast so micro services power our production studios and serve customers via our AWS infrastructure and we do all of this using lightweight micro services power by Redis we think this is all pretty badass [Music] [Music] I hope you enjoyed this presentation if you're interested in applying these ideas or have questions please feel free to reach out thanks for watching Carlos so we have plenty of time for questions here I would be interested to know how did you handle the transaction management when you interact with different micro services so so in our particular case our micro services are focused largely on moving data from you know the bikes in real-time you know into our into the our infrastructure there isn't really transactional aspects per se in this case right because you have a class that's running you have a bunch of participants in the class and so this is all pretty segregated in that sense right so it's not something that's come up but we our shop uses a combination of like RDS and and and MongoDB so you know transaction capabilities there along the way but that's uh my question about transaction is basically interacting with micro service two of them succeeds and and couple of them failed or sending information but my lines are about like how about the two requests that sits forward and then two-phase and I have to go back and take the data so this is a essentially the switch Y at the service law at our level this which wise and it will just go to another micro service if the micro service was unable to respond to and process the requests like I have a question about them my cursor can you speak a little louder sorry hello yeah I have a question about the micro-services and so you explained the diagram about the monolithic operation the normal vertical scaling kind of an application so in that kind of an application we had the advantage of the speed right so you do ten operations sequentially but it is done in a particular time but when you expand it to a micro service authentication happens first in a micro service component then the next operation happens so the trip taken to complete the complete operation takes a long time to complete so do you think it's an advantage to go towards micro service so so not all operations are sequential I mean I understand that sure you need to have a user authenticated before other they perform other actions on the system so you yes to that but but you do have the ability to process these you know request you know at scale and so you you know these are these are a case of trade-offs right if you want absolute speed my recommendation is you know you write assembly or your write C++ as I did for most of my career worked on game development right you know you can focus on speed if that's your goal if your goal is sort of a flexibility of time scalability then you know then you have to allow for things to possibly taking longer but you can optimize for them keep in mind that you know you can look at your at your infrastructure and determine where the operations can be sped up with what kind of hardware a good example is that um you know you can divide the request coming in from the outside from your infrastructure obviously as micro services communicate with one another that's a that's at a much faster speed than going over public internet and so you know you really have to look at each of these key pieces and um and you you know you have a keen observation which is hey when we had a monolith right at least things were like you know super fast you know but then you lies with your monolith that well you know you need to you know add a new algorithm or you need to change something and now you need to scale the whole darn thing together as one piece and allocate resources for that one piece as opposed to being having the the benefit of granularity so I software development right it's all about trade-offs any more questions thank you thank you [Applause]

Channel: Redis

Views: 16,401

Keywords: Redisconf18, redis, redis labs, redis enterprise, microservices, nosql, data, databases, Flywheel Sports, developers, devops, tech

Id: z25CPqJMFUk

Length: 49min 5sec (2945 seconds)

Published: Fri Jun 01 2018

Reddit Comments

[u/RamesesLabs]

Great presentation and we will definitely be looking into building with Redis more. Keep up the good work.