High Availability and Load Balancing

hi everyone thanks again for being here my name is Ruslan Khurana and I one of the open sips developers I've been with the project for more for about eight years seven or seven years now I'm about to make eight years and today I will be talking about high availability and load balancing at the edge of your open sis platform it will begin as a short story but will easily evolve to a more technical one we'll be talking about DNS probably all of you know what DNA is and how it works we'll talk about VA vrrp anycast and we'll see how all this link up in order to offer high availability and low balance balancing to your platform so probably most of you are already using this kind of scenario where you have a bunch of a storage server here you have open tips in front of them just at the border just to ensure access to your customers in your entire network and to your gateways so there are multiple there are multiple advantages advantages of these scenarios for example you can impose security here you can ensure aggregation of IP so you don't have to present to your gateways all these IPS you simply present open six IP and that's it everything will will go smooth however some of you might have seen crashes or something happens on this link and open ships goes away how do you solve how do you sort this out well the the solution is simple you simply put more two or more instances and you're set up if one goes down then you have the other one however is not that simple because you would ask yourself which one do we tell our customers to use because here we have two instances though therefore we have two that's where DNS comes in handy as I presume all of you know how DNS works but I'm going to show you like a brief presentation so basically DNS is used to resolve fqdn domains num names into IPS actually into services it's considered one of the first distributed application so this it's is being has been developed in 1985 so basically most everybody knows about it and most of the clients already supported and it has several types of queries like a an apt air as our V PTR whatever so there are many but we today will be talking about this to how DNS works is very simple so imagine that you have a customer you have your platform registered at example.com you just tell your customer he's your user ID here's your password registered to example.com what is your customer doing what is the SIP client doing actually it first does on an apt our query to find out the the services that this example.com platform has and here you can see that it has two different services one is over UDP one is over TCP which one we choose well that's where this field comes in basically this is the DNS precedence and tells you in case you have multiple entries first use this one if this one doesn't is not working then choose this one the second one so the lower the presidency is the higher the possibility to choose it is so as you can see this is this somehow offers kind of a failure mechanism in case this one this server is not reachable then use this one so this is the domain the first step into failover next we take this this response and do on SRV query what does it sarva query tells us it tells us where the services so in this case you can see that you have you have example.com serving UDP queries at sea panic at example sip on that example calm and subdued at example.com again how do we choose them there is also a precedence here but in case these two presidents are equal we go to the nest next field this is the wait the DNS wait is based to do a sort of load balance so everybody will if for example if these two values are equal there are 50% that some of the queries get here and some of the queries get here furthermore we just take this response resolve it on a query to find out the IP of the server in this case this is the IP of example.com and simply forward the sender registration there this sounds really nice right let's see how this works in practice unfortunately in practice it doesn't go that's des smooth why because let's say customers are or clients usually register for longer periods like one hour if it's a desktop or a hard Worf hard phone if they use if they take one hour to register what happens if the machine goes down if the Registrar goes down well it happens that every single inbound call that reaches the server it won't get to the I won't get to the client so inbound calls will not be available of course our BIOS will be available because the client implements the Philip if the client implements the failover mechanism it will reach the server and being sip being based on UDP usually there's there are not that many mechanism to find out whether the server is a port or not some of the clients the SIP clients are doing pings and if the server does not respond they simply failover to a different one but most of them are not and most of them don't have TCP support because if they would have had TCP support they would have immediately figure out the server is down and move to a different server so as you see DNS does not provide any kind of high availability if one server goes down is very it simply has failover abilities so we somehow need to register to share the register information with the second instance so and I have here an example what how that works so imagine that you have your customer you have a DNS a DNS response that resolve to two different servers and you have a replication mechanism between these instances now when the client registers it let's say it chooses the this this server so it sends its registration information and these guys stores it and then forwards it down to this to this guy so now both instances know the same thing both instances know where the green guy is therefore if this goes down and the call counts an Enka invite comes then you know exactly where the green guy is because you also have this information right okay so that being said we need we need the application layer to be able to to also support this kind of replication mechanism otherwise failover and high availability wouldn't work correct and there are multiple types of multiple possibilities for doing for doing replication for example you can simply replicate all the traffic mirror or the traffic to all the other instances well this is not that efficient you would probably have noticed you can also use a shared database to share to have the shared information do have the share the registration so both the two instances can have can point to the same database and if one stores some information in there the other one can simply read it however this means that you have to do a real-time database query and that's not efficient either you probably don't want to do that that's why in opensees we built a different replication and native replication channel that is built on top of the class or module so basically you can have as many instances as as you want in a cluster and you can share the information in a very reliable and very efficient way within that cluster this is offered by the clustered module and the binary interface module but Don talked about it in the previous presentation I am NOT getting into too many details because I've already presented this last year okay what kind of data can we share well as we've seen we need to share a user the user location or the user registration information we also need to share the ongoing or active calls the dialogues as they are called in zip we might need to share some rate limiting limitations so for example if you want to have a shared concurrent limitations so both servers need to share the how many calls each of them have in order to have a global limit on the entire platform so that's that's a kind of replicated data that you need and basically you can share anything using some custom script functions so that being said this is this looks very promising right however in practice is not that promising because why because usually here there's a big wall of firewall what does the file do is basically when this guy registers it opens your app in whole that maps from this guy only to this IP so if this if this sisters goes down and an invite comes in it will be blocked by this firewall because that's how that's how that works so we need a solution we need the mechanism to move the IP from here to here right because otherwise this guy del firewall will not allow any inbound traffic without without the client opening of a new pinhole to the new IP that's what vrrp does so vrrp is basically it comes from virtual router redundancy protocol and is used to monitor multiple instances and elect a single master node that master knows received of the virtual IP and all the traffic with that is sent to that virtual IP and there are a couple of solutions very stable ones ones they are very old like VR PD or keep alive both of them do the same thing and both of them work on top of the VRP protocol the way it works so imagine that you have multiple instances and the first master the first master node since periodically sends pings to all the other instances instructing them that he's available he's alive you can handle traffic if one of the machines are missing those pings he starts a new election algorithm and a new node a new master node should be elected so this sounds this sounds promising this regarding the previous problem this might sort out because if as I said all we need to do is move the IP from one side to the to the other instance and we are set because we will be able to break the firewall in practice VRP although it works really well it might become had a very a bit difficult to configure at first why because by default vrrp is not used to monitor services it is used monitor machines so if you for example only one to monitor the Copeland's if service then you have to write your own scripts your own your own monitors - and your own instead your own treasures and indices that indicate you whether open zips is up or not and in case something is wrong you'll have to find you'll have to give up the master the master label let's say another thing that you have to take into consideration is that it vrrp require support from network equipment so you have to be in the same multicast domain in order to be able to in order to be able to move the IP that's because multi does it because there are piece based on the multicast protocol and this multicast protocol is not but it is not routed in the Internet it's only routed in local network areas so basically you can't move on IP from one location to a different location unless you have a basically or land between these two but it works I mean we've been using it for a while and you were really really good but is it enough let's see so we we have the previous example where the client is sending a register to this open zips both instances have the register information and well in order to keep this pinhole alive you'll have to send periodically send pings either from here or from here we usually send it from here that's because not all the clients know pinging so that's why we are doing it so we are sending pings and everything is fine this pin hole is opened however this guy also has the registration information so he also thinks that he's he needs to ping so he will send pings from here however he doesn't own the virtual IP so we'll probably use a completely different interface that will be blocked by this firewall and that's not the only problem imagine that you have like dance then instances of opens is what if all of them would have pinging with her with her would have tried to pink actually him you'll basically have a lot of useless traffic so that's why we also need the application layer like open zips to somehow partition the data and we need a mechanism to tell open ships to stop acting on behalf of the replication data so you we need to tell this instance that you don't need to ping this guy because there's somebody else that's doing it also when something goes bad and this guy goes down you need the mechanism to tell him now you can start to ping it because now you have the virtual IP now you are allowed to go through the firewall and maintain the pinhole right all of this is already done in open ships using some simple and my commands so you don't have to go into too much trouble like because there are other solutions like adding IP tables rules to prevent this guy from pinging but there's still some processing some useless processing that's done we can we can get rid of that using open ships and some simple mi commands there are also other pitfalls like the virtual IP is only pinned to a single IP because there's only one IP that can there's only one machine that can get traffic for a specific or for a single IP right you can have it assigned to multiple machines but if you do that you will probably need to disable the traffic otherwise the the machines will start to take each other's IP DRP a ARP table will go nuts and you don't really want to do that so you have to you'll have to set up some IP tables rules to prevent others machines that already have that IP from processing traffic again traffic goes to a single machine so all the others machines are just and by so this might look like waste of resources right you can also do a master master so for example if we have two instances you can do a master master setup but you will need multiple vrrp instances and you need to play with the with the iptables rules in order to move them to the it always disable the iptables rules from the active machine and enable it on the backup one so it's it's not hard it's not that hard it's doable we are using it but you might run into some into some issues and again the machine be the IP being assigned to a single machine is not that efficient because you can really do load balance you can only simply send registration like zip traffic to one single machine and that's it all the others are standby or can process traffic for that specific IP and that's why where any cost comes in handy right so what does any cost mean or do you know there were a couple of presentations about that this year too so basically you have the same IP assigned multiple machine actively assigned multiple machines and each machine can act on behalf of that IP so each machine can process traffic from that IP this is a very very easy way to balance traffic usually it's usually implemented over routing protocols like BGP so imagine that you have one IP assigned actually I have it here so you have imagine you have three instances all of them have here towards your customers a single anycast IP right so all of them have assigned on their machines some the same IP there's a router here that advertises route to all three machines so when an invite comes in it is sent it isn't based on a distance metric because that's how routing works based on a distance metric is it is sent one of these machines one of these instances and from that on you can use unicast here or any caste whatever but I will I am using now unique a step is to keep it simple now we have we have an invite that goes to an answer a server that's reserved very response since a 200 ok here which is further sent here right how do we route this 200 ok how do we how do we make sure that the 200 ok goes through this instance and not through this one well here we have the unicast address so we the reply will always go here it won't we can go to any other to any other instance not at least not in this scenario because we are using unicast type is here so the call is responded and we send here the the 200 ok this guy is happy somebody answered his call and he's sending an ACK well now the ACK according to the routing protocol and because here is a unique anycast IP the ACK might get on a different server well we get here how do we know to send it here well that's simply record routing mechanism so this is really simple and now this guy is happy this guy is also happy every so the call establishes correctly however what about this guy this guy didn't get the ACK right so he will start wondering what happened with my ACK I he's not able to complete his dialogue because he didn't get the ACK that's why we need a mechanism to share dialogue information between all these instances so this guy when he starts the dialogue he needs to tell this guy well there's a there's a new call from here to here and these are all the metadata when did when this guy gets the ACK it finds the dialogue the call it completes it changes its state life saying that the ACK was received and needs to tell this guy well you have the ACK has got here so you are free to you should also update your metadata right and everything works again when a bike comes in it might be sent to a different instance right well we also route the by according to record so we will know the by will end up here but again this these three guys should replicate this by event to all of them so that all of them can close their active dialogues and everything is sorted out right how can we replicate as I said earlier we can use the Placer module which is doing replication replication in a very very efficient way all we have to do is provision each instance in a cluster and tell the dialogue module to replicate that information in the cluster we only replicate dialogue events so it is very very efficient so we don't replicate like entire messages or stuff like that and it's done using the binary interface again I won't be I won't be talking about that that's not it let's see why so good going back to our example we've seen what happens if this guy initiates a call right what happens if this guy initiates a call let's see so let's see it chooses the same instance well it will choose the same instance why because here is a unicast IP and the invite goes here to this customer however when it sends a reply the reply is not necessarily landing on the same server it might go over here and we will get it here well again based on the via the transaction module we know exactly where to send the reply well this raises some completely different problems because in this case now we need to replicate a replicate a completely different kind of data we need to replicate transaction data and that's not that easy what happens if we don't do it well this guy hasn't received 200 ok yet so what happens if you don't receive a reply you start transmitting so this guy starts sending again invites to this guy this this receives the invites and starts sending 200 ok probably here and basically you you get a lot of useless load so we need to find a way of replicating transactions however this is a bit problematic why because a transaction contains a huge amount of data as opposed to the dialog for example you'll have all the color and call information you also have that in the dialog but you all you have all the branches information so for example if you are doing something like a parallel forking and you have like 10 destinations you probably have you needing to store and replicate information from 10 different destination as opposed to the dialog in the dial in a dialog you only have one only the successful leg right so it's it's it's way more information and also for transaction you need to to replicate the full messages why and that for each leg you need it because if one of the if one of the call is do not reply then you need to retransmit again so in need all all the instances need to share all these information so you end up sending like a few kilos of data for every single call and this is not something that the data is very volatile for example invites so this data needs to be replicated within in less than a second for example you might receive a provisional reply immediately after you an invite also for by transaction the boy transaction is less less than a second because you send a bye you receive the 200 okay and that's it but you still have to replicate this information to all the other instances so it's it's very very is a very traffic consuming and processing consuming issue to solve and we also need to coordinate who's in charge of rare transmissions because otherwise we'll end up in the same in the same situation with the registrations right everybody will try to return submit everything and basically will flood the client that's how we we try to sort this out in we given a thought and try to sort it out and we'll end it up in with two different solutions one of them is using a lightweight we call it lightweight anycast so imagine that you you have here an invite that's sent to the client and go through this through this open sips instance so this guy stores the transaction information and that's it he doesn't replicate anything when a 200 ok comes in you simply replicate the event the metadata here saying that ok I've got 200 ok you don't have to do any more read transmissions here and simply pass further the 200 ok well so this is this looks like a very simple solution the transaction is stored only where the request first gets in only event or state changes are replicated so you don't really replicate all the messages as I said earlier and it's also everything is replicated using the cluster ER and B in binary protocol so is everything is very efficient there the pros for dissolution is that it's very fast it is lightweight so you don't have to exchange huge amount of data and information but the cons is that if if the initial machine crashes he's the only one that knows about the transaction so basically everything will the transaction will be lost and the call will not be able to be answered another thing that that's a limitation is that you don't really you can't really do anything on those replies because here the reply you know all the transaction information you have it here so this guy doesn't really know how to what kind of in what kind of changes to do on this on this message he simply realize it back and that's it so as I said earlier this is a this is a good solution it sorts out the problem but not entirely that's why we we've thought of a different solution like a full Anika's solution where we again we have the same scenario and invite God gets here we store the transaction information and send it further to this to this guy the $200 key gets here but now instead of only replicating data we replicate the full sip message okay when this guy gets the fulsome message it will send a 200 okay here and everything will be sorted out right so this guy doesn't need to know about the transaction is all this guy so again the transaction is stored only where the first request gets in this since the system's receives all the messages related to that transaction from whenever wherever they get so any instance can get a reply it will simply forward it to the to the main instance and messages are sent again over the efficient binary protocol this allows us to build more complex scenarios because having the having received every time the reply where the transaction stands you have all the information to perform any changes on that message so basically this is how we offer a full failover and high availability solution however the cons are that that there's more information to exchange because every single time we don't we don't export only the event but the all the message received all right so let's let's do a small wrap-up so again DNS although it provides of failover mechanism it doesn't really have any high availability considerings although it also provides some load balancing mechanisms and is supported by most of the SIP lines well actually not all of them but most of them there's also some nice DNS extensions for example like geolocation you can pick up a server based on the location of the IP so you can get the server that's closer to you and do failover in case that goes down and so DNS is quite a very quite an interesting and a must have solution in your network also VRP is a very nice solution that we've been using again for a while and it is used to move by piece from a failed machine to a new one this way we provide high availability we can have multiple machines that share the same IP there's also we also have solutions to solve the network constraints however it does not offer very good load balancing abilities that's why with most of the time we need any cast to provide both load balancing for internal load internal edge proxy and failover in the high availability support it is relatively easy to implement however it is connectionless so this means that all the applications that share this that use this mechanism need to somehow have a method for to replicate data between them luckily open sheaves already does that so bottom line in order to have fully over in a fellow very enabled and high available platform you have to have your club your clients they need to fully support the DNS logic the CBN is logic we also have to control your network layer in order to handle or either vrrp or any caste supports but also the application layer needs to be aware of the data it uses and it replicates and this is done very easy by open city using the cluster module and in the binary protocol you also need the application layer to coordinate which instances are responsible for the replicated data because otherwise if everybody thinks that he's in charge of that data we will end up with huge huge loads of useless traffic and we don't want that so as I said there is no there's no secret tool there's no no single secret tool to build a reliable and efficient work platform all you have to do is study solutions put them together and see and achieve what you want thank you very much thank you very much husband does anyone have a question that they would like to ask all right can you tell me which module is used for that which one which module we will use for that for any cost yes so we we will this will be built on top because it's not already available these are only the solutions that we discussed and we are going to implement they will be available in in the transaction module so basically I think everybody uses the transaction okay what what will be the name of the module so that we can see it but TM open 6 TM TM right ok all right other hands over here ok so this is not generally available yet yet yeah we we we it it's actually ongoing work it's I'm going we're going right I'm just also curious about the message passing done at the transaction layer right how do you do that well basically we use the binary protocol which is used as a generic message passing protocol between open sips instances to pack the information from memory and send it to all the other instances why is it is that dense via some kind of message passing right right which one do you use it's not it's a native one is built by us oh okay all right other hands here we go have it has any thought been given to replicating presence across yes that's the next on our list it's not it's not there yet but it's all it's on top of our list and is there any thought about distributing the any cast with servers in separate physical locations that's something that has to be taken care of on the network level so if it's if the network layer so if they were public loves it right okay thank you back there all right simple question what's the timeline when it'll be available any clue it will I think it will be available on the next release and that's going to be next next year in May that's that's when we usually do releases so it's probably be available by the end of the year I'm not sure how that's that's true but it will probably be available the next release will be in May so that's when you have it stable there was also one question here is anyone else questions hands in the air yell at me if I'm not looking at you all right actually two questions one is well what's your target for having consistency across all your nodes with your with this protocol and then secondly if I recall right on me any on there on your full anycast URIs replicating the message back to the main node right but what happens that main node goes down with the other nodes that weren't aware of didn't seem like a head state right something yeah we probably need to build a different logic for for the transaction but only the so we don't need to replicate the messages this this way because the messages will still be here and will be retransmitted by this guy actually yeah you need to if the one of the left goes down right after the invite shipped I mean that there's no yes somehow change the owner of the transaction so in order to change it you'll probably have to replicate the entire transaction over all right but how long were you how what's your goal a sip transactions especially if it's very local it's very quick so what's your what's your target forgetting for getting message from one on one box processed and in the tables on the other box you know from a transmission are we talking you know what were you talking a few milliseconds so are we talking ideally it will be less than a few milliseconds idea but that depends on the network is not like something that we can control all right other hands no hands no hands all right wrong very much [Applause]

Channel: Official Asterisk YouTube Channel

Views: 2,473

Keywords:

Id: ICbSMMcTh6U

Length: 39min 21sec (2361 seconds)

Published: Mon Mar 26 2018