Building Distributed Systems with Node.js and ØMQ

okay anyway as I saying so thank you Jesse for inviting me to come out as he mentioned my name is Jim Wilson I go by Jimbo a lot of the time I'm author of nodejs the right way it's a pretty good book I think you should buy it it benefits me financially if you do so please do that on github on Jimbo JW where you can find this talk it's called node zmq talk is the name of the project so you can follow along or use the examples if you want and i'm on twitter i'm hex Lib if you're into that sort of thing the title of this talk is building distributed systems with no js' and 0 mq which is kind of auspicious in my practice for this talk before giving it just now it takes me about 50 minutes to get through the content I'm going to do my best to stick to that if it goes a little long I apologize but it shouldn't go much longer than that so before we get into the content I wanted to just share with you kind of where work where we're headed first of all I'm really excited about both node and 0 mq because of the makeup of this audience and the fact that it's a node Meetup I don't pretend that I have to sell you on node but I might have to sell you on 0 mq so I'm going to talk a little bit about why it's important and why I like it and why you should - we're going to skip how to install it because you can do that later if you really care and then after that to introduce 0 mq first I'm going to give you a native publish/subscribe implementation very quickly and we'll talk about kind of four major problems with that implementation that will lead us into how to do pub/sub with 0 mq and then we can talk about other messaging patterns like rec web rec rep which is request reply diva router which is for parallel request reply and finally push-pull which all of those will talk about if there's time at the end we can do questions if you have clarifying like quick questions I'm happy to take those during the talk but if they're very deep philosophical how does this work or why why does this matter those I might punt till the end if it's something in between then help you so why is your own queue as your own queue is for making reliable resilient fault tolerant distributed systems it's for connecting programs and having them chat to each other it's very low latency it's not a lot of overhead it has a philosophy of being event-driven and non blocking which as you can imagine has a lot of overlap with the node philosophy it provides you rich patterns for interacting with programs and it offers scalability now that buzzword is pretty loaded you know how do you make something scalable and the answer is not just throwing more stuff at something right if you add more boxes to a failing diagram you just get more fail right so the way you achieve scalability in the 0 and Q Fawcett philosophy is by reducing the cost of adding additional components to your distributed system so the philosophy of the 0 and 0 mq is trying to really really just make it as low cost as possible to add additional parts to the distributed system and we'll see how it does that as we go along to understand where 0 mq fits in the scheme of things here I have a kind of a high jacked OSI model so the OSI model if you're not familiar with it as a way of stratifying kind of network behavior at the lowest level level one is the physical layer that's actual wires or the microwave transmissions or whatever it is that's sending bits slightly above that is the datalink layer which would be like your MAC address and above that is the IP and above that is TCP so 0 mq lives at the layer above that in the application presentation and session layers of the OSI model along with other things like SSL and HTTP SMTP which is a for email so that kind of gives you an idea of where you live so you wouldn't implement something like HTTP or rest on top of 0 mq because it's a peer with 0 mq at that same level of things right you can use one or the other but you wouldn't implement one on top the other you could implement something very much like HTTP on top of a zero and cue infrastructure but it wouldn't be true HTTP because of zero and Q's interaction with the underlying sockets a quick show of hands who in here is use something like ActiveMQ or RabbitMQ something like those a good number of you yeah it's nothing like that so it's it's lower level than that so though if you wanted to develop RabbitMQ or if you want to develop an active mq 0 mq is the kind of library you might use to do that it's the the lower level thing right so like not-it's fairly low level it's still what it still requires you to do some work as I said we're going to skip all of this installing stuff the gist of installing is that you have to install the binary first and then do an NPM install so I'm going to presume that everything's happening you've already done the NPM install and we can get into the naive pub/sub implementation so I presume because of the nature of this meetup that you're all fairly familiar with nodes so I'm not going to spend a lot of time on kind of node basics but I do want to walk through a pubsub implementation so that I can tear it to shreds in this beacon application my publisher just emits heartbeats and my heart beats our JSON messages that include a process ID and a timestamp and the timestamp is just a JavaScript millisecond since the epoch timestamp so let's look at the publisher real quick okay so here's my publisher code it's a pretty terse the first thing the top is we're going to use Atma script harmony strictmode and another thing that I'm going to do here that might be unfamiliar to you is I'm going to use Const 4 constants instead of bar and I'm going to use let 4 variables instead of R if you're not familiar with those things Const is exactly what it sounds like it's a constant variable you can only assign to it once and let is like bar except instead of being scoped to the nearest function its scope to the nearest block so I use constant let because I can instead of bar but you can replace var with these and everything will work just fine so in my naive publisher implementation I'm going to require the the node net module set up a connection pool and then use nets create server to create a server that listens for connections every time I get a connection I want to push it onto my connection pool and when it closes I want to find out what index it is and remove it from the connection pool ok so that's just kind of basic overhead of handling connections and then on an interval of a thousand milliseconds or one second I'm going to create this data blob consisting of the process ID and a timestamp and then for each connection write out that data right so this implements my beacon I'm going to publish this message every second lastly a little bit of banality I have to do is tell the server that I created to listen on port 5 4 3 2 if you use Postgres PI 4 3 2 is the default port that it runs on so if you try to run these and they tell you you can't it's because there's a conflict with the port numbers just pick a different port number and everything will be fine so anyway I listen on port 5 4 3 2 and a minute message that we're listening so I'm going to get that started in a terminal and I hope that you guys can see this in the back I tried to make it pretty large I have Nets net beacon Parliament okay so I'm I'm running this using node harmony the Harmony flag is what allows me to use that constant let nomenclature I believe Const is baked in now without the Harmony flag but to use LED you have to use it so anyway the Harmony flag lets me get away with using that constant let I mentioned earlier and I'm listening for subscribers and presumably this is trying to iterate through an empty connection pool and there's nobody listening so let's fix that so here's the subscriber code this is even shorter again I'm going to use strict and again I'm going to pull in the node native net module and this time I'm immediately going to create a comment that connects to port 5 4 3 2 and whenever I get some data run it through json.parse and log a message that I got that pid' with a nicely formatted version of the the timestamp that I got let's go ahead and run the subscriber and unsurprisingly I get heart beats every second right and I can shut that down by hitting ctrl C and if I start it back up I get heart beats again right so this should surprise no one and in a fairly short order using only node core code I was able to make a pub sub system so what's wrong with this turns out there's a lot of wrong with it and I'm going to talk about those things next so let me go ahead and shut this down hope you guys don't mind I'm doing live demos instead of just talking about it okay so I have four things that I've kind of selected to talk about for what's wrong with it the first one I call listener bias listener bias is the unfortunate feature of this distributed system because I have a distributed system now I have publishers and subscribers and they're distributed and they can they chat listener bias is the unfortunate characteristic that if my listeners not running my whole system doesn't work right and I can demonstrate that by trying to start up the subscriber without having the listener running if I try to start up the subscriber right away node tells me econ refused the connection was not allowed right the TCP port is not being found it's not listening so nothing happens everything fails so that's listener bias suppose though that I got everything started correctly if I got everything started up nice and neat and I'm receiving subscription messages and then I kill my publisher you'll notice that the subscriber also shuts down right so this is the second problem which is called fault intolerance even if the system start state was good as the ongoing running distributed system runs things happen sometimes the network has a hiccup or processes die in this case my whole system shuts down my subscribers don't start back up even if my publisher restarts itself automatically the subscribers don't so as a developer I have to now plan for this I have to plan for the connection not starting up correctly and I have to plan for failures as they happen so my work is getting is getting harder all the time but that's actually not all the problems that there are there's two more the third problem I wrote down is called leaky buffers leaky buffers is best shown by looking at the code real quick so if I look at the publisher the publisher says connection dot writes data so I'm relying on this connection to just send out these bytes which happen to contain a JSON message and then in my subscriber code go back to the subscriber code I presume that I'm getting a whole JSON message and then I'm sending it to json.parse but nothing in the stack nothing in the stack below my code guarantees that I'll get exactly one JSON message on a message boundary it could be that I get a half a message it was still a legitimate message but it comes over to events instead of one event if it got chopped up on the network on the way to my subscriber so I now as the developer have to implement buffering code I have to bring in a node module or write my own code to say I'm going to buffer up these data events then look for some sort of boundary character to know that that is a full message then try to parse it then move on to the next one right so again now my work is getting harder as a developer trying to make a resilient system the last problem the last problem that I have listed here is directionality so directionality is more of an architectural concern it's pretty clear it's pretty clear in the publisher that this publisher is going to create a server and it's going to manage a connection pool all of that is really baked in so when you have a publish/subscribe system it's fair to say that most of the time typically your publisher is going to be the stable endpoint and your subscribers are going to come and go I mean most of the time that's probably what you want but it doesn't have to be that way I mean why couldn't I make a system where my subscriber is a stable piece of my architecture and publishers come in and publish a message and then leave I maybe I want to do that in this code I would have some serious refactoring to do to implement that change because it's very clear that the connection handling the connection pooling is all part of the publisher code if I wanted to move that over to the subscriber and have this and have the publisher connect I'd have to do a lot of kind of refactoring shotgun surgery as a way of saying it so there's this marriage between the direction that messages flow which is the publish to the subscribers and the direction that the TCP connection happened to be established in right and I might want to separate those but in this code I can't there they're kind of I've made that decision early on and I have to kind of stick with it or do a bunch of refactoring okay so finally let's look at some 0nq code and how it does all of that better if I can manage my scroll bars okay so let's look at a publisher with 0 mq this code is actually a little shorter for reasons that we'll talk about I'm also going to use strict as before this time instead of pulling in the net module I'm going to pull in the zmq module this is available through NPM and I've already installed it so there it is then I'm going to create a zmq socket of type pub the return object of this is a zmq socket object that's kind of your principal means of interacting with 0 mq in node land and I'm going to save that to a variable called publisher then on interval of a thousand milliseconds or one second I'm going to create a message object including the process ID at timestamp just as before and I tell the publisher to send the json stringify version of that message so note that what's missing here what's conspicuously absent compared to the old publisher code is anything to do with a connection pool I don't I don't care at all about the connection pool I don't care who's connected to this publisher at all as the user of this library 0 and Q handles that for me behind the scenes lastly I tell 0 mq to bind the tcp endpoint port number 5 4 3 2 on any available interface which is what the asterisk is and log a message that I'm listening for subscribers so I'll go ahead and start that up in a console and it's listening for subscribers and let's look at the subscriber code you guys are really quiet like disturbingly so for a roomful of people okay so here please ignore the fact that I use double quotes for this use strict I thought I got all of them but I left one in apparently I'm going to use strict again pull in the zmq modules before and create a zmq socket of type sub so again I'm getting a socket object you'll have all the same methods that my publisher socket had earlier except this one's a sub I have to tell the subscriber to subscribe and I have to pass in a string when I'm passing the string that I'm passing here is a prefix string so in my case I actually do want every message so I have to pass at the empty string if I didn't do this 0 and Q would give me no messages so this is a little bit of bureaucracy that 0he requires but it's because if I wanted to instead of using a just a simple JSON serialization I could use some other serialization that uses a relevant prefixed string and then have 0 mq at a low level internal to its binary filter out which messages I get but in this case I just want all messages so I say subscribe to everything whenever I get a message I say json dot parse that message and out output to the console the pid' along with a nicely formatted timestamp and lastly I tell the subscriber to connect to TCP local host port 5 4 3 2 so it's we can sub and to no one's surprise this receives beacons on a one-second interval if I shut it down it stops receiving beacons obviously and if I start it back up it receives beacons again okay so that's all fine so let's talk about the four problems that we mentioned earlier and see if 0 and Q does them better so the first problem that I had mentioned earlier was listener bias so that has to do with the start system start state of the system so I'll go ahead and shut everything down and now we'll try to startup the subscriber and see what happens so it looks like nothing happened right but an important thing is that it is actually running and it's waiting for beacon messages and behind the scenes 0 MQ is trying to connect to that endpoint it's on its own back off schedule so I didn't have to implement a backup algorithm or anything it's trying to connect to the publisher and when the publisher comes up it'll eventually work so let's go ahead and start up the publisher side and the subscriber starts getting messages right so as a developer I don't have to worry about the start state of the system it'll handle that for me if I kill off the publisher the subscriber keeps running and it just stops receiving messages which is what we would expect and behind the scenes it's trying to reconnect to that same port and if I start up the publisher again I get messages again the the pit numbers here are very similar but you'll see that it's actually a different process ID that's answering to the latter published calls because it is a new process ok the third problem was leaky buffers so in my subscriber code it seems like I'm cause it seems like I would be doing the same thing I was doing before in the TCP socket version I'm just calling json dot parse on the data but this is not the same kind of event that the TCP socket was the TCP socket event was just giving me some characters of data and it was my responsibility to know whether or not it was on a boundary but in this case 0 and Q guarantee is that when you send a message it will either deliver the whole message or nothing right so I don't have to worry about the buffer problem because this is actually a message event that will either have the whole thing or not now I could still defensively code around JSON I mean I've agreed with myself in my distributed system that I'm passing JSON messages it doesn't stop a bad actor from sending something other than JSON but zero and Q will at least guarantee that it's going to deliver the whole thing or nothing at all so that that leaky buffers problem is resolved and the fourth problem was directionality so as I mentioned towards the beginning of the pub subsection of this talk I'm getting a zmq socket object that socket object has methods connect and bind and I just either call connect or bind depending on what I want to do so switching this around if I really did want to change the direction of the connection would mean I would tell my subscriber instead of connect to something I would tell it to bind an endpoint and then in my publisher code I would tell it to connect instead of bind right so I could flip that around without kind of deeply RER connecting all of my code right so that brings us to the end of the publish/subscribe section the next messaging pattern I want to talk about is request reply one thing I want to mention so at the outside of this talk the talk was called building distributed systems with with node and 0 mq and I hope that I'm giving you the building blocks for doing that doing higher level design is probably outside the scope of what I can talk about but we will be doing clusters as you can see very shortly but before we get to clustering I want to talk about req rep so req rep is for request reply much like htd much like HTTP a request comes in and reply goes out but unlike HTTP there's a lot less overhead it's really just a much tercer way of sending messages so the first thing that we're going to do here is set up a reply socket which is going to listen for incoming requests so here's my reply code if I can control the scroll as before we're going to use strict we're going to pull in this your mq module and this time I'm going to create a zmq socket of type rep for reply but I like responder instead of reply er so I'm going to save it to a responder variable and then when my responder receives a message because it's a responder the type of message that it would receive is a request and what I want to do is parse out that request data which I expect will be JSON and I expect will contain a Pitts logout that I received your request and then send back my response my response is going to be again process ID and a timestamp JSON encoded and lastly I'm going to say responder bind to local host 5 4 3 3 so new port number and log a message that I'm listening for requesters so let's get that started in terminal what did I call it time rep that J is okay so that's listening for requesters and let's look at what a request looks like so the other end point of a rep would represent the same size because it takes about the same amount of overhead to set each of them up again we're going to use strict again we're going to pull in these zmq module this time we're going to create a socket of type wreck for requester when the requester receives a message it's going to be a reply so when I get a reply I want to parse out the response and then note to the console that I got a response from this process ID and what its state stamp is I'm going to then connect to CC 4 5 4 3 3 which I know to be listening for reak replies and then I'm going to log a request so send a request and sorry console dot log is sending request for time and then requester dot send and here I'm just sending a pid' instead of a timestamp because presumably my requester wants to get a timestamp not send one so let's run that so sending a request for time and then nearly immediately it gets response back so my reply ur receive the request and then send a response back and everything's fine so you'll notice that my rec Jaya's program is actually still running right which might seem odd like why would my requester be still running if I've already sent the request and the answer is because there's nothing stopping me from sending more requests maybe I want to send more requests so behind the scenes 0 mq has already established that TCP connection and we'll keep it alive and that's why my node program is still running until I shut it down it's going to stay alive which also keeps the node program running if I kill it off and run to begin I get another response and I get another response right so request reply is pretty straightforward but it happens in lockstep so what I mean by that is as you can see in the reply code my responder when I get a message I'm telling my responder to send the response back right so you might you might worry like what if I got multiple requests at a time how would how would it respond how would it know which one to send back to I don't have any connection I don't have any connection object that tells me this particular request gets this particular reply and the answer is 0 and Q doesn't let you get multiple requests at a time if you're using a socket type of rep the contract that 0 mq enforces for the rep socket in particular is I will give you one request until you give me a reply and until you give me a reply you'll get nothing else right so that might seem counterintuitive if you're really used to node running lots and lots of asynchronous i/o and asynchronous processes simultaneously it might seem a counterintuitive and almost blocking to only get one request at a time and queue up others but there are reasons for doing it which will become a little more clear later but fortunately you don't have to live with that req rep is not the only way of doing a request reply there's also these things called dealer and route which are for doing parallel requests reply and so next I'd like to show you how to make a cluster that behaves just like this rep but does it by setting up a pool of workers and distributing requests to them so this picture is pretty big on that screen and make a little smaller okay okay so this is a picture of the cluster that I'm going to show you the code for a little bit this code is the longest code of the night so that's why I'm doing a picture first on the left you can see my client program that has a rec socket a socket a 0q socket of type rec it's going to connect to a TCP endpoint which is handled by the master process of my cluster my cluster that that TCP endpoint is going to belong to a router a router is like a parallel rep it's like a reply ER that routes the incoming requests and can do multiple set in in parallel the master process also has a dealer which is like a parallel req and it's going to connect to an IPC socket inter process communication it's basically a UNIX socket 0 mq supports a bunch of different socket types we're only going to use TCP sockets and IPC sockets here but it has a bunch more and then each of the workers that are spawned by the master process are going to use are going to have a rep and behave just like our rep did before but they're going to connect into the IPC socket okay so let's look at the cluster code okay so as I mentioned this code is the longest of the night but it's not too too bad I don't think so use strict as before and we're also going to use the 0m q cluster module have you guys used cluster before raise your hand if you use the node cluster module only a few okay that's okay it's not too it's not too hard if you forked a process and see it's not like that at all so keep that in mind we're also going to pull in the 0 and Q module as before I had just have a hard coded worker count of 3 you could set this to be whatever you want or determine it from for example the number of processors that the computer had that might be a useful number and as I mentioned there's gonna be two end points one is a TCP endpoint that's going to listen on localhost five four three three that'll be four external connections for the incoming requests and the worker end point will be the internal IPC handle which here i've called a filer dealer but that's it can be called anything this is a holdover from the book I copy pasted it as a copy paste error so I apologize but it's just an IPC if this will become a file named Philo dealer data IPC on the file system and it'll be the the UNIX socket that we use so in node when you're using the cluster module if you say cluster is master that will be true if this is the first program that is started with the cluster module and then if you start workers is master will be false so this code that follows is what the master code will do it creates that parallel router that I mentioned and binds it to the external endpoint because I need I need my router to be accepting internal request from the requesters and I create a dealer and by net to the worker endpoint now whenever the router receives a message what I have to do is pull out the frames of that message so we're not going to talk too deeply about frames but zero I'm cute but I'll give you a little taste of it the way that 0 and Q handles routing such that it can give a response back to whoever made the request is it uses this framing a call so it's very low overhead as everything is with zero MQ but it frames out in such a way that you can nest frames so if you had multiple layers of your architecture where you had multiple you know like a fractal of request replies you could you could get back where you were going we have to pull out the frames and send them over to the dealer and then like likewise whenever the dealer gets a response back from one of the replied the workers we have to flip that over and pass it back to the the router right so that's what these that's what this little bit of magic does it took me a little while to figure out how to do that I haven't looked at what these frames contain so if you're interested you could run this code and like console that log the frames that see what's in there then whenever then the cluster module we want to know whenever a worker comes online so we'll just log its process ID to the console and then we want a fork using cluster dot fork worker count workers right and then each one of those when they start up that cluster that is master will be false and so then they'll execute the code in this else block so whenever a worker comes online it's going to set up a reply err just like before connect it to the worker end point that's that arrow in the diagram that connects back and whenever it receives a message this is just like the code from before parse out the request and send our send a response so let's run that kill us so we're going to run the clock rep cluster my three worker processes have come online and if I run let's clear this out if I run the request again I see that I got a response from 3107 nine which is the first worker and if I kill this and run to begin kill it and run it again I get responses from the second and third workers I'm respectively okay so what that is is behind the scenes that dealer is dealing out the request to each worker round-robin style so another thing I got for free out of this code is I didn't have to implement a round-robin request response dealer 0 and Q provides that for me ok so that's that cluster there's actually going faster than I thought maybe I'm talking too fast oh well sure I assume the colossi running running sure so the question was is your own Q is not running any sort of server in the background and the answer is yes I'm not running any sort of extra process it's really just a binary library 0 and Q was originally designed by some really smart guys who have been working in messaging for a long time and it was actually first done for see in like financial applications when very low latency very low overhead was kind of critical for doing these fast financial transactions in C where they're passing structs around and there's in if you read the documentation it'll talk to you about how to deal with null terminators so in C your strings usually end with like slash 0 but in 0 and Q you don't want the overhead of that extra slash 0 and it already has a length field in its framing protocol so there's instructions on how to deal with the fact that your C strings come back without a null terminators and like putting the null terminators back on because it's kind of low-level it has since been ported to a bunch of languages it's available in Ruby and PHP and Python and Java c-sharp a whole bunch of others note obviously but it's it's it's like running a server because it is a very powerful binary but it is not actually a separate service it's just baked in right so all of that queueing that it's doing when it's queueing up messages that haven't been delivered yet all of that's just part of its binary operation the last pattern that I hope to talk about is push pull push pull like pub/sub we mentioned the beginning the messages only travel in one direction so in publish/subscribe publishers send messages and subscribers receive them in push pull the same is true the pushers push messages and the pullers receive them but unlike pub/sub only one puller will get a message and if there are no Polar's available they'll be queued until one is so publish/subscribe doesn't guarantee the delivery of anything if you publish a message and there's no subscribers nothing gets delivered but in push-pull if you push a message it'll wait around until somebody's available to pull it down so the code looks actually quite a bit like publish/subscribe so the push code here again I'm going to use strict I'm going to require zmq and create a socket of type push I'm going to keep a job counter which starts at zero and then on an interval of one second I'm going to push one job and that job is just going to contain a process ID and a job count so this is if I was making some sort of like work queue I was going to have some sort of push or pushing a bunch of work and pullers that do the work I'm going to bind to any available interface on port 5 4 3 4 and listen for a zmq pullers so I'll get that started real quick what I call it work push yeah okay listening for pullers so it's actually pushing once a second messages on the queue and those are just getting queued up so until I switch over and start a puller we'll we won't see those messages but they're not getting lost and the pull side of things looks very short and much like you might expect we're going to create a zmq socket of type pull and then whenever we get a message parse it out and then just log we receive this job number from this pit right and we're going to connect on startup to port five four three four so if I switch over work pull you'll see that right away and received a whole bunch of messages because they were all queued up it took me about 50 seconds to get over here so that's how many messages had queued up in that time and now it's receiving once a second the Polar's just pulled them down okay so the very last push-pull thing I want to show you though is what happens if there's multiple pullers so I have another cluster application this one's simpler than the other one though this one again we're going to use the cluster module and the 0q module and have three workers the master is just going to log out that the workers came online and then fork three workers so the cluster the master here only exists to spawn three worker processes it's like the simplest possible cluster and then the workers do exactly what the pullers did before they fire up a socket of type pulp and then on message log it out to the console and then they all connect to TCP port five four three four so let's switch over and we'll clear this out and now start the pole cluster okay let me let me kill this so what I want to show you is that during the time that I was talking a whole bunch of messages got queued up and processed number three four ten which was the first one of the workers to start up pulled down whatever was a hundred and twenty of them before on the one hundred and twenty first message another worker was available to pull it off right so from then on if I start the cluster again from then on you'll note that eight six seven eight six seven its round-robin distributing them to the pullers but note and zero and Q are both so fast that whichever worker happens to start up first will pull down virtually everything that's been queued up before the next one even gets a chance to get into in the cycle okay so this is in the book I call this a first joiner problem right this is a higher-level architectural concern our low-level architectural concerns have already been dealt with we have a fault-tolerant system that can start up in various states it handles the leaky buffers problem and I can architectural change who connects and who binds but now I have a higher level problem which is if one worker starts up it grabs all of the work before anybody else can get any of it right so there are solutions to that problem too like we can have the workers announce I'm available and then have the pusher say only send messages when X number of workers are available but that's kind of like we're now at another level of distributed system right so at least so zero and Q's job in life is to get you over that first number of hurdles all of that kind of low level message passing business and then now it's up to you the architect of the distributed system to handle these higher level concerns so we'll kill that go back over here and that's all that's all the content I had so we talked a little bit about y0 and Q's is important we talked about the pub/sub pattern the REC rep pattern the dealer router pattern by showing a cluster and the push-pull pattern I thank you for your kind attention and I'll happily take questions Jesse that's like however they dealt with that you make 60 seconds later see part of my first question okay Thank You Jessie I was I was queuing up in my mind all of the parts of your question so that I could repeat them so the question was with higher level message queue systems like rabbitmq or you mentioned another one I iEARN MQ I've never heard of that but okay with so the question was with higher level systems they'll do work guarantees if I send a message and it's been accepted but then it hasn't been acknowledged or something how do i how do I know that it finally got it I believe 0 and Q has features for that but I'm not as familiar with them there's a good O'Reilly book by the creator of 0 and Q on 0 and Q it might talk about it but I'm not sure offhand I'd have to look it up why did you use their cricket sure so the question was in my professional experience have I use 0 mq and why is it better than those other systems that we've already discussed so the answer is and I know this is sacrosanct because this is no js' in the wild I've only used 0 mq in the context of demonstrating these patterns to the book because you asked about my experience and because my experience is in the book I'm going to talk about the book I'm not only I didn't pay Jessie at all to ask that question I promise in the book in chapters kind of 5 onwards it's a lot about HTTP and doing rest and restful stuff I wanted to talk about messaging patterns as aside from HTTP and 0 and Q gave me a good excuse for doing that I also was curious about it and it gave me excuse to research it for the book but have we used it instead of RabbitMQ or something I haven't done that in production so I hope I get invited back to the Mita Oh sure so the question question was specifically with publish/subscribe if you're pushing messages do they get queued and where do they go with publish/subscribe in particular if there's nobody subscribing it they just get dropped so publish subscribes guarantee is anybody who's subscribed and will get the message anybody who's not you're out of luck with the other with the other messaging patterns where there is a queue involved like like if you were doing a reply socket and you're just taking a long time replying it will queue up I believe 0 and Q has it has a bunch of settings I don't know how to set them in node but presumably there are api's for doing so but you can set things like the high-water mark which is when I get to this level start dropping or when I get to this level throw an exception or do something or throw an error so I think there are ways of programming around those higher-level concerns about dropping messages but I haven't I haven't used them in node in particular Oh yeah absolutely so the question was is 0m q cross-platform if I had a Java program could I communicate with node absolutely it is so the bot it's a binary as it was written in C so you have to install the binary first well since we're talking about it I'll show you the installation process if you're using a Mac you can brew install zmq and it'll install the binary it gives you those like dot Esso files those like binary modules if you're using a boon - it's a little more complicated because you have to build it from source but it's not too bad and to test if it work you can just say man zmq that's what I do yes you can use it with Java or anything else because anything else will just bind into that binary module and then do its thing and provide some sort of language appropriate abstraction on top of that so in node we get zmq socket objects if you're using C you'd be calling functions with struct pointers you know anybody else up if you had the need to to maybe put it put something to long-term storage if your if a cube got too big would there be a way to catch it you mentioned it you throw an error which goes oh maybe just throw it to some sort of storage if you really wanted to have a really strong sure so the question is if you had a really strong queue and you wanted to deal with messages overflowing what can be consumed online and store them in a more durable way how would you do that so higher-level messaging systems like RabbitMQ have facilities for that where you say here's my queue and here's the backing file system that I want you to push those things to zero in fuse lower level than that so it would be on you as the developer to monitor the number of outstanding queued messages which I don't know how to access offhand but I presume you can access and ask it how many cubed messages do I have and then start throwing those storage so you again if you were going to be implementing something like a rabbitmq are ActiveMQ server 0 and Q gives you the tools for doing that but but it's not going to necessarily store that stuff for you another question best marketing like benchmarking against wet Kafka so I'm not familiar with that technology but I haven't done any benchmarking the question was have I done any benchmarking I haven't but there are good benchmarks available on the 0nq website I believe they have some white papers operator so on the so so on the on the read the manual learn the basics page of 0nq it links to this 2011 CERN study that compares it to CORBA ice thrifts whatever RTI and cupid and then has a white paper of their conclusions so I haven't read the white paper but other people have done those benchmark analyses anyway thanks I'll be hanging around for a little while buff's talk to you smart a pet

Channel: jimbo

Views: 19,074

Keywords: ØMQ (Speech Topic), Node.js (Software), Distributed Computing (Conference Subject), JavaScript (Programming Language)

Id: zgDjaJdAB9c

Length: 48min 1sec (2881 seconds)

Published: Mon Jul 07 2014