Splunk : How Search Head Cluster Captain Election Works

okay in this video we will talk about how how the searched cluster captain election process works okay so if you remember from my previous video I have created for searched clustering I have said that the captain election is a dynamic process right that means it it happens automatically when the triggering condition happens for the captain election okay so to start with what what will do is if you see I have I have this for searched members right such as 1 2 & 3 & 4 which basically forming a searched cluster okay so so this if you remember from my previous video and I talked about the add and remove the searched cluster remember I created this searched for as well right so among them this searcher 1 is a captain so if I if I just show you the current configuration over there okay so we will start with from there then we will try to see how this how this whole captain election process works okay so perrolli what I will do is when it is logging in so maybe we need to log in to some other search heads as well so I will just I will just make it ready so in the meantime let me let me draw let me draw this set up over here okay so what I will do is I will just try to create this for search heads over here okay so this is my searched one this is my searched - this is my search tree and this is my searched for right so let me draw some text over here as well so this is such it one this is our searched two over here this is such a tree and this is such it for over here searched three and the searched for over here okay so among them one is the captain so if I if I just run so this we are in currently in searched one over here so I'll just sudo will clear the screen okay so if I just go to Splunk home bin folder okay CD /opt Splunk bin okay now if I just run this command Splunk show this H cluster captain sorry is each cluster status over here okay so it it should show me the current setup password here okay so if you see it over here we have these four circuits in our cluster and among them our captain is our search it for now currently okay so so let us let us write it down over here as well so take so I will just make this guy this is our current setup now okay so now we will first try to discuss what actually triggers the captain election process okay and then we will try to see live as well so I will just make it as some space over here and then I will go over here so if I just write it down the triggering condition triggering condition for election okay so these are the these are the triggering condition for for election over here okay first thing is when the captain goes down okay so current captain goes down over here okay so that means something happened it just sat down or something happened over here this this will definitely trigger a captain election process okay now it may happen that one odd mode searched member okay Amy P er is out of the network out of the network due to some let's say due to some network network partition what partition happened okay network partition happened okay because of that one or more searched members are out of the network they are not able to connect with other members over there in then through that network right so this will definitely trigger a captain election process okay now as as the as this network partition occurs and it it created or it basically triggered the capital election process it may happen that you fix the network right and again that particular member is is again with sync with basically able to connect with other members in that particular Network right so that will that will again trigger the another searched captain election process okay so that means after fixing after fixing the network partition okay so again if the member in two point number two is able to connect to other members okay other members again a search it captain captain election orchid occurs okay again the capital election of course this is another triggering point over there okay now there is another triggering point for a searched cluster captain goes down by itself okay now in Splunk searcher cluster what happens is if there is not majority of that searched members are up or basically functioning the captain automatically goes down okay and that basically triggers the captain election process so that means captain captain steps down okay captain steps down because it detected it detected the that majority of the majority of the searched cluster UST cluster members members are not functioning okay members are not functioning so this this could be the probable triggering condition for the search area lustre captain election process okay we will see some of some of the examples today okay now what does not trigger the captain election so this is also important to know so what does not trigger the election okay now now here basically so here only one point I would like to mention is like suppose in non cluster a in non cap tain member went down okay window not due to not due to this this network partition but some other other condition like not we that is worth mentioning over here not due to network network partition or network issue network partition okay so if a non cluster non captain member goes down okay in that case it generally does not trigger the captain election process under Al Andalus that that member is out of the network because of this network partitions and all okay so we talked about the triggering conditions for the captain election process okay so let us now see how it works okay because we are saying it's a it's a dynamical election election happens right so currently our search at four is our captain right so let me shut it down okay that means I will just stop the Splunk over here and let's see how the cluster is behaving so for that what I will do is I will go to such at for now okay I think I have not opened the search at four so let us SSH to our search it for now okay we logged in over here we will first sudo then we will go to Splunk home bin folder okay now I will just shut it down so dot slash Splunk stop okay so I'm just stopping my Splunk over here so our captain goes down now right so we basically we basically created this condition over here the captain goes down so let us see in other circuits how it's working so I will just run the searched status now again if you see now it is saying encountered some error while trying to obtain the searched cluster status okay fail to proxy call to the current searched cluster captain this guy over here 90 hatchet for was our previous captain right so it goes down so this search it cluster are not able to communicate with the captain so at this point till the captain election works this searched cluster this searcher cluster members only have there only they aware about their current environment they do not have any idea about the whole cluster now because they are not able to communicate with the captain and captain manages everything over there right so this this captain election process occurs after 1 to 2 minutes I think it takes a certain amount of time because internally something happened ok which I will be showing you I will be maybe by drawing the picture I will try to explain that to you ok so until and unless that is happening so this searched cluster member functions like like their own ok so let's see let's done this status again just to see whether this captain election happens or not still if you see it is still not happened now let me try to run that command again ok now if you see it over here there's there is automatically captain election happens our current captain is searched to now right and nowhere we have searched for because that that's that particular stuff is down over there right so so that means the new captain election happens and current captain is subject to now what happened in between when when the capital election process works or basically how it works so let us try to understand that ok so whenever this guy searched cluster 4 was down right so to do that what I will do is I will try to make this one ok so I will say ok I have to go over here I will just try to create a small one so that we will we will understand this search it for is down over here ok and well what I will do better I will do one thing I'll just directly put this one a small circle over here okay and I will make it relate over here so that means our searched for is down okay now the capital election process starts whenever searched for is going down over there right so during the capital election process what happens is what Splunk do basically is it set up a random timer in all of this all of this searched cluster members okay so if I just try to draw a timer over here okay so it sets up a three timers over here okay now now I will show you okay after after this one I'll show you another adult stuff over there let me let me draw the trimer first okay so okay so so let's say this is the first first timer over here okay so okay three three timers we are we are setting it up for each and every searched over here okay so let us make it better okay so three time our Splunk sets when when when this this particular captain goes down over here okay so so what happens after that after that these timers are random timers over there right so now these timers will be off now because this is the random timers over here right so the timer which is goes which completes first four of any of these members that member you will be basically say okay I am I will be the new captain over here so please vote for me okay then based on the majority the the capital election process happens now if you seek our current captain is searched - that means internally it must have this guy completed before okay so if I if I just make it as a separate color so this guy this guy completed before this guy timers actually finished before compared to the other two guys so this guy says okay I am electing myself as a captain so please vote for me so in that case order it happen the other searched members okay will will vote for will vote for that one now here here is the biggest catch over here okay so now if you see we we have four members cluster over here right so when when we say majority that means at least at least there will be a three member should be present or should be functioning for that majority two works right majority does not mean the majority of that running searched members okay majority means majority of the overall members so to get the majority we have to in count the all the members of the searched cluster as well so that's why I am saying three because even the even though if if we have let's say another searched member goes down which will see eventually okay what will happen in that case will have only two searched cluster member which is functioning right that is not majority because two is not majority out of four right we need at least three over there right in that case even though the searched cluster captain election process triggered the captain election process will never be succeed in that case because there is no majority of the no no majority of that such a cluster member exist in the system over there okay so so now generally when when a time our best searched cluster member cont or basically it says and the captain now so other members generally comply with that one and they vote for that particular member okay and and the captain obviously vote for itself so that's why this this majority works over here okay so we saw when the captain goes down so this is our current captain now right so this is our current captain now so let us let us go back to our documentation over here so what what I have seen like the captain when captors goods goes down no first first let us see if I if I just bring back this search it for over here okay so what will happen over here if I just do the Splunk start over here okay dot slash plunk start okay so if I just start the Splunk now what will happen is really trigger a captain election process definitely not no because this this search it for is not the current captain now even though it's goes down because of some some maintenance work or something not not due to the network partition just simply going down and coming up it will it will never trigger the captain election process over there right and let us see how it will behave it will just simply join this this cluster then as a member of worthier right as I have shown in my previous video as well when you talked about the adding or removing that such a cluster member so when when we just simply shut down Splunk stop or Splunk shut down a member right we are not deleting the configurations the cluster level configurations from that instance right so whenever it is coming up it should be automatically added to the cluster as well over here if you see when I just ran now this such a cluster status our search to is still our current captain and four also joined over here automatically okay so now let us try to see another scenario over here when I said a captain automatically goes down or steps down when it detected that majority of the searched cluster members are not functioning over here the the majority I was talking about this is the one right because in that case the captain election process failed over there okay even though even though that captain automatically goes down the captain election process is also failed because there is no majority over here okay so to do that again I will just turn off this this search at for now okay so I'll just plunk stop here so search it for is going down now if you see this this this should not trigger the captain election process as well let us let us try to see now this is a good scenario okay still it is still that it is not propagated that searched for is went down so let us let us give it some time after that it should say it should not list searched for as a member of this cluster let us try it out again it takes certain amount of time before before it do that so let me try to run it again if you see now this is this is showing our searched forests down over here right so that means our our searched forest down now now let let us try to down another member of over here okay so which is a non captain member over here okay so let us try to see it over here so our current captain is current captain is two right so let me try to shut down the searched three over here okay so for that I will just copy this guy and I will do it over here okay so I just I just go to our search a tree this is our this is our such a tree here okay I'll just sudo to sue I will clear the prompt over here okay I will go to Splunk home bin folder opt Splunk bin folder here okay and if I just do this plunk stop here okay so let's see now what what is happening to this particular cluster okay so again from the searched one I will just I will just try to run that same searched cluster status now okay now let us see the captain should automatically go down okay and we should be receiving an error over here because among these four searched cluster members right two members are already down that means the captain has detected right there is there is no majority of the members is running that means ideally in the democratic process it will just go down or it will just step down as a captain over there okay let us try to see if it is happening over here it takes out an amount of time when when this such a cluster three will be down right in that time that particular stuff will be working let's wait for a couple of wait for one minute I think so if I run that search it cluster status now if you see over here it is saying failed to do a box proxy call to that such a cluster captain previously our such a two hours hours kept kept in right to zero for right 21 to 0 for all right this also says this node is not the captain of the search head cluster this is the searched one right and we could not determine the current captain because even though we did not we did not shut down the current captain but because of the majority of the members are down that triggers the captain to goes down as well over here okay so so now until unless we are we are having a majority of the member the the captain cannot function at all the cluster cannot function at all because to to elect a captain right we need the majority of the members who vote over there when it when I talk about the majority of the members it is the we have to we have to take it into consideration all the searched cluster members over here not the running members over here okay so so we basically turmoil there they are peaceful harmony over here right so let us try to let us try to Bing search it cluster three over here and let's see whether whether it is automatically coming into life or not okay well I will do is I will start the search at cluster three now okay so let us wait for a couple of minutes okay our searched cluster three's up now let us try to see the status of the current cluster know if you see that the moment I I ran that such a cluster three some interesting things happened over here okay first of all our our searched cluster searched member two was the previous captain stepped down over here right so that means a whenever I bring DUP this searched cluster member three over here right such a three over here it triggers the captain election process because there is no current captain over there right so that's why that's why the it elected searched one is the current member now current captain now okay so that means if I just mark it down over here so this this becomes the current captain now and it follows the same process the timer process over there okay to elect a captain over there okay so so one is now our captain and and as our majority of that members are present over there so captain election process so was successful previously it was not successful because of there is no majority of the members over there right so and I just wanted to tell you another stuff over here as well okay it may happen that when majority of your members are down right and you still want your cluster to function so there is a way or there is a concept called static captain over here okay which we'll be discussing in next video we will see in that case to to overcome this challenge how to set up the static captain and and and and just make this cluster function for a for a certain period of time and then how to move from a static captaincy to the dynamic captaincy again those things also we will try to discuss as well okay so hopefully this video is helpful okay see you in next video

Channel: Splunk & Machine Learning

Views: 1,691

Keywords: splunk, how to, captain election, cluster, sh

Id: aAKZhCXksXU

Length: 26min 6sec (1566 seconds)

Published: Thu Sep 19 2019