Exploring the replication and #sharding in #MongoDB - #Database Tutorial

in this session we will be exploring  the replication and shouting in MongoDB   few words about me I am working as a database  consultant with PTM I joined Pythian in 2014 in   the open source database practice and since  mid of 2016 I am working as a lead database   consultant for MySQL and MongoDB practice  from 2015 I am a MongoDB certified DBA at   the 600db a certification in this session we  will be speaking about what is a replica set   in [ __ ] database how the application works  what other application concepts later we will   see about the replica set features and the  deployment architectures we will speak about   vertical versus horizontal scaling then we'll  continue to a Charlotte cluster in MongoDB what   are the components of Charlotte cluster we  will see about the different Shard keys that   we can define in Charlotte cluster what chunks  represent and at the end we will discuss about   hash two versus ranged based char link you can  ask questions any time you can interrupt me so replication and MongoDB is maintaining the same  data set from multiple processes that's the   basic definition of replication in any database  primarily we do want to maintain multiple copies   of our database because we we want to provide  high availability by maintaining multiple copies   the replication provides high availability but it  does not scale so the application itself is not a   scaling solution it's primarily for availability  and by some points it can increase our reach   capacity because moving from a single node we  now have multiple copies of our data where we   can read from with MongoDB the replication  goes in in the following way there is client   application that speaks to the driver that  is connected to the replicas set initially   all the write operations go to the primary  note the primary note stores all the changes   to the data into a collection called operations  lock and once the changes are recorded into the   operations log then asynchronously all of the  entries in the operations log are replicated to   the secondary nodes in the replica set once  the changes are replicated to the secondary   members in the replica set both or or all of the  secondary members in the replica set will copy the   dialogue into its own operations lock so by doing  that it's allowed the secondary node to use sync   source another secondary and that's by default  in every replica set configuration the setting   chaining call-out is true by default and once  the secondary sees that there is another node   that is not lagging mode in 30 seconds it can  initiate the sync source to another secondary   you can disable this setting if you want  to always replicate from the primary node one important setting for the replica set  members in MongoDB is the operations log and   the operations log itself is a kept collection  that keeps rolling record of all the operations   that modify the data in in the databases it's it  important it means that you can replay all the   operations in the Oh block as many times as you  want at the end it will have the same result it's   important to mention the default o clock size for  UNIX and Windows systems by default it will take   5% of the free disk space so if you are starting  a replica set with small disk size probably the   OP log collection will have small size and that  will be important for your replica set if later   you have right operations that will modify your  data or even a high amount of insert records the   operations log is the log that maintains column  PureApplication window will be initially for   the plank of replica set if we come from the  MySQL world we may think there are two nodes   for maintaining a replica set but that's not  the case with MongoDB we we should always start   with at least three nodes and form a replica set  because by having three nodes we can have majority   for election when there will be one node down  starting a replica set is you can install the   batteries from any package or even from tarballs  unpack on the three nodes it's worth mentioning   that all of the three nodes should be able to  speak to each other in the same network whether   it will be local a network or one network so once  we have the boundaries installed on all of the   nodes we run RS initialize on the first node and  once we do that we have the primary node for our   replica set now it's interesting that we already  have a replica set but it only has one node once we do that it's very easy to set up  a replica set in MongoDB because what we   what our next step is is to add the rest of the  nodes and we are doing that by just running the   command RS ad and specifying the hostname and  the port that the host is listening to again   everything in a replica set is maintained  from the primary node we we run our s ad on   the primary node and it already knows what  the secondaries are and from the same time   the secondaries will start copying the data  from the primary node each replica set can   have up to 50 members and seven members can  only vote in elections there are different   configuration options we can have arbiter node  we can have priorities euro note hidden nodes   delayed node we can manipulate with the right  concern and read preference folder replica set the arbiter node is only part of the replica  set and it it votes in elections it does not   code copy of the data and this is mainly used  when we need extra copy of our data we can   have primary and secondary we don't want third  copy of our data but we need a third node that   will be voting in the elections so that's why  arbiters are used or we can use arbiter if we   have deployment in two data centers each data  center have two data nodes and now if one of   these data centers goes down we need an arbiter  node that will participate in elections and one   of the other data centers will elect primary  node priority 0 node is member from the replica   set that never becomes primary if we set the  priority to 0 to any member in the replica set   then we should expect that the node will never  become primary and high or it will participate   in elections and it has votes so this is useful  with the same example is is the arbiter node we   can have three data centers but in one data center  we don't want any node ever to become a primary   and we do that by specifying the priority to 0  similar to priority 0 node there is option to   set hidden node but this is a little different  while priority 0 node will be visible to the   driver and the applications can direct traffic to  the priority 0 node hidden nodes are not visible   to the driver and by that our application can not  see the copy of the date so we should be cautious   while setting hidden nodes hidden nodes can also  never will never become primary node because the   the priority of the hidden node is 0 mainly  the use cases for K denotes is we said hidden   note that the application will not see but we can  query manually by doing reporting or backup jobs   on on those nodes the late node is a node where  we can set the replication thread to delay from   the primary node by certain amount of seconds  the delayed node must be a priority 0 node so   it does not get elected as a primary node this  is important in elections because as the node   is delayed we don't have the latest copy of the  data and we don't want this node to be elected   as a primary whenever there is a failover in the  replicas set however this node will participate   in elections and it has votes so again with like  the the hidden node this node can serve reporting   jobs or it's primarily used for backups one  consideration for the late node is that we   should monitor the block size and how long our  application window is so that we don't override   o clock and make the delayed node out of a block  entries it depends not very common but from the   late node if you run some delete operation  on the primary you have at least maybe hour   or two to get that data back from the from the  delayed node so that that's why they are useful one important setting for replica set is the  right concern this setting is mainly managed   on the driver and it allows the drug the  application to care for response from the   replica set on how many members the right  succeeded by default this is only one member   from the replica set the primary note but we  can change it we can have W two members which   means that when we do right on our replica set we  are expecting at least two nodes from the replica   set to acknowledge the right however this may  not work as expected because even if we have a   right concern of two which means that we can  expect two nodes to acknowledge the right or   the right will fail with MongoDB if we have a  timeout on a right concern with two that does   not provide a guarantee that the right did not  succeed so the right may be successful on the   primary node but we did not get acknowledgement  from another member that the right was successful   so the option is only to get acknowledgment from  as many nodes as we expect to acknowledge the   right however the rights may be successful  on the primary and it will get replicated so which we should be we should expect that  this behavior is possible with MongoDB another   setting for replica set is the read preference  this shows how the read operations are routed   to the replica set members by default the driver  will send all the read operations to the primary   node but sometimes this is not what we always  want we can change this property and we can   make the setting primary preferred with primary  preferred by default all the reads will be sent   to the primary node but in case the primary  is not accessible the driver will route the   reads to the secondary nodes the other setting  is if we set secondary suite preference and by   setting secondary suite preference the driver  will always send the reads to the secondary   nodes similar to the primary preferred we have  option to set secondary preferred and by that   the driver will always try to send the read  operations to the secondary but in case the   secondary is not accessible it will route the  read operations to the primary and there is   another option for setting nearest in the read  preference and with nearest the read can go to   any node depending on the latency between the  driver and the nodes so the driver sends pink to   all the nodes and based of the network latency  decides was the nearest member and then it it   can send the read operation to the nearest  member starting from MongoDB 3.4 there is   mark staleness seconds property which allows us  to set read preference to secondary but only if   the secondary is no longer than certain amount of  seconds legging behind the primary if it's legging   more than 300 seconds for example don't read from  the secondary go to the primary it does not so it   routes by round-robin yeah so the replica set  allows us high availability but at some point   of time we want to scale and we have our data set  the working set grows one thing that we can do is   scale vertically we can add more CPU we can add  more RAM even we can increase the disk size or   have faster disks but at some point of time we are  limited we cannot scale our Hardware indefinitely so what MongoDB allows is a horizontal scaling  or shutting and primarily the horizontal scaling   is not something that one would you be invented  but it's this is a method for distributing our   data across multiple machines and having our data  spread across multiple horizontal partitions how   this is how this fits into MongoDB everything that  we already discussed about replica set is turned   into a Shard or a partition and with MongoDB we  have a router and config servers so each component   is part of the older Charlotte cluster and it's  important for the MongoDB clustering as mentioned   previously the Shard or the partition should be  a replica set and that's important because we   want to keep multiple copies of our data in a  single partition the the other important part   of the of the Charlotte cluster are the config  servers and that's where the metadata is stored   for for the Charlotte cost the third part is the  router or the mum ways that's the interface for   for our cluster and all of the operations that  go from the application to the Charlotte cluster   should go through the [ __ ] s we shouldn't do  any changes to our data directly going to the   Shards however we we have that option but we  should definitely know what we are trying to   do by modifying the data directly on the Shard  everything in a Charlotte cluster should go only   from the routers or from the [ __ ] s once we have  our replicas set the deployment to the Charlotte   cluster is having the the config servers and  having our mum ways the garage the Shards itself   are edit through the always interface simply by  running the command SH a Shard and the sharp name once we have initially a Charlotte cluster with  one Shard then we can add as many shots as we   want so the Shard or a replica set in MongoDB  cluster is a subset of the data and it should   be always it should be used as a replica set to  provide high availability every every cluster has   primary shard for each database when we create  a database in a Charlotte cluster the database   initially stored on one of the Shards so we may  have five shots but initially when we create the   database it will be stored on sharp one so that's  the primary shot were the database resides it's   important to mention that all the collections that  are not Charlotte will reside on the primary shaft   and this is important because after some time we  may be experiencing high load only on a single   shard from our clustering we may be wondering why  why this is the case but all the non Charlotte   collections in the database will reside on the  primary shard for the database so starting from   MongoDB 3.4 all the sharp members in a Charlotte  cluster must have the option shard server in the   configuration file and must be running by  default will be running from 4 to 7 0 18 so the question is is it possible to have  more than one shared collection yes we   can have as many Charlotte collections  as we want each collection is shouted   separately will we'll get to the  shouting you know in a few slides the other important part of the Charlotte  cluster is the config servers the config   servers are storing the metadata for the  cluster and that's where the information   for each document in the database on  what Shard is located resides other   than the metadata for Charlotte cluster the  config servers also store the authentication   configuration and that's stored in  separate database called admin database one important change starting from MongoDB  three-point-four is that the balancer that   is balancing the chunks in a Charlotte  cluster now resides on the config server   primary node and the config service by  default will run on port 2 7 0 19 the   last part of the Charlotte cluster are the  [ __ ] s nodes the the longest nodes are   the interface in the Charlotte cluster  and they they don't have any persistent   state the Montes updates the state whenever  there are changes in the cluster metadata and before MongoDB 3.2 the long ways also had  the balancer which is responsible for in   the data between the Shards in the shadowed  quest all the operations that go from the   application to the Charlotte cluster  must go to among us because only the   Mongoose has the information what document  on what Shard can be found and in what chunk so we have our shouted cluster deployed and ready  and we have our database we have our collections   now the next step is to actually Shard our  collection and Shard our date first for Sharding   a collection is enabling shouting on a database  level and we do that by just running the command s   H dot enable shouting and we specify the database  name once the shouting is enabled on a database   level the next step is to shut the collection and  there are two ways for shutting the collection we   can shut on a range based shutting on we can Shard  on a hushed page shutting but before we move to   shutting may be the most important step that we  must do in in a shuttered cluster environment is   to choose our short key the sharp key should have  all of these properties so that we can be certain   that our data will be distributed evenly of all  the Shards that we will not have hot Shard and   we will not have huge chunks that later will be  we did later the bouncer will not be able to move   across the shots so the first option for a short  case the short key should have a large cardinality   that means we should have enough unique values for  the sharp key so that there will be many splits   from minimum to the maximum value in the Shardey  and that we will have enough chunks that later on   will be balanced across the shafts so the question  is if we have a unique key is that good candidate   yes that is excellent candidate so for example  if we have a user ID and the user IDs are you are   unique in our collection then probably that that  key is quite good candidate the other option for   good Shard key is that it should have low low key  frequency what that means is even if the Shardy   has high cardinality and many unique values maybe  there will be some key that has too many values in   our collection so with the example of a user ID in  the users collection maybe some users have most of   the users will have hundreds of documents in the  collection but some users might have hundreds of   thousands of documents in the collection and if  that is the case probably we should think about   compound Shardy and not only the single Shardy  and the other important setting for a short key is   that it does not have a monotonically increasing  value like thumbs temp or object ID but this   property is more for range based sharding while it  will work good if we are doing crushed by shouting once we define our short key and we shut our  collection the documents are logically grouped   in chunks there is no physical division of  our data in in the cluster but all of our   data is logically grouped into chunks and the  chunk is simply a continuous range of sharp key   values that reside on a particular shot it has  the lower bound of the sharp key inclusive and   the upper value of the sharp key is exclusive so  for example if we have a sharp key on some number   starting from 20 to 30 20 the value of 20 will be  in the chunk but 30 will the value of 30 will not   be in the chunk and so on upto the mark scale the  default value of chunk in a shadowed cluster is 64   megabytes which means that once the chunk size  gets approximately to 64 mega right there will   be a split and the the split allows the next  chunks that will be formed from this splitting   to grow up to 64 megabytes and so on so the the  cardinality of the key is important because if we   have few unique values for our short key then  we are limited to have multiple chunks in our   system and for example if we have only 10 unique  values for our schottky then in our system there   will be maximum 10 chunks and those 10 chunks can  recite each chunk on a separate Shard but we are   limited with 10 Shards on our cluster even if we  have 20 shots we are limited with distributing the   data across those 20 shots because our cluster  cannot split our data in more than 10 chucks so   that's why it's important to have a good short key  cardinality however the chunk size by default of   64 megabytes can be changed even if you want  to change to a lower value of if we want to   increase the way so the first option for shutting  is range based sharding that's dividing our data   into continuous raging that are determined by  the shared key values so what that means is if   we have numbers from 0 to 100 the Shard will  the shutting will be distributed from the min   key up to the monk [ __ ] key and probably the  documents that have closer values like 25 and 26   will be probably reside on the same Shard so  this is important when we do find operations   in our Charlotte cluster because if we want to  scan and find the documents where the value is   greater than 25 and less than 30 probably this  operation will be run against a single shot and   we will not have to query each shot individually  and then much there is the result set so the the   range based shouting will work if we want to  make sure our find operations will always get   the result set from a single Shard or we should  be aiming to get the result set from a single   Shard so so the question is the difference between  chunks and Shard Shard is a section of the cluster   that has horizontal copies of the data so if  you look at the Shard in a cluster and okay the shot itself yes copy of the data from the  cluster that is distributed while in the shot   itself the data that resides in the shot  itself is split into multiple chunks okay opposite from the range bass shouting we  have option to shout our collection with   hushed bass sharding and we do that by running a  command to share the collection with crushed key   it's important to mention that we do not need  to compute the hash from our application we   just say cache our use cache page shutting for  this collection on the user ID and the MongoDB   database will calculate the hash key for us and  will store the data into appropriate chunk we   don't need to have some key in our document that  is already cached and then to store that key in   the collection in this case gives like object  ID or thumb temps that monotonically increase   are good candidates for hashed key but we should  be aware that with harsh taste shutting probably   the find operations that we run will have to be  sent to multiple Shards to find our documents   and that's mostly for range based queries where we  search for example give me the documents where the   user ID is greater than 25 and less than 30  probably in that case our query will be run   against multiple Shards unlike the case when we  have range based shopping some of the Schottky   limitations are that it must be ascending index  key or at least compound key that exists in   any document in the collection if we have single  document in the collection that does not have the   short key then we cannot Shard that collection by  using that key the short key can not be multiple   index cannot be text index or give special index  also it cannot exceed 512 bytes like we already   mentioned or we didn't the Shard is immutable so  once we set the short key for a collection and we   have document with the short key we cannot update  the document and change some value of the short   key we can only delete the document and insert the  document with with noushad key and also the short   key is hard to change it's not impossible but  it's hard because we need to have a logical dump   logical backup of our collection then drop the  collection insert the data and shut on noushad key yes now then at this time there is no option for  changing the Schottky just by running a command   collection dot shard key change or something  like that maybe maybe there will be in the   future but at this time it's not and and that's  why it's important to choose a good char key for   our collection at the beginning we have to dump  from [ __ ] s that's because we want to dump all   the shards maybe there are 10 maybe there are 20  shard we have to dump everything and that requires   downtime one other limitation for the shard case  is that if we have shouted collection we cannot   add unit unique index on any other key only the  Schottky can be unique index in the collection so   to hear the summary or for the Charlotte clusters  and the replication in MongoDB we should always   use a replica set with odd number of voting  members and that's important because if we have   even number of members and half of the members  fail then we will not have a majority to elect   a new primer for dedicated functions like backups  or reporting we can use hidden or delayed members   in a replica set configuration if our data set  fits into single server probably it's best idea   if we keep the environment unsharded unless our  data set grows and that we cannot longer handle   the data size in a single replica set then we  should starting off shot but the shutting itself   at complexity and it's more difficult to manage  than having a replica set and running our data   in a in order pick set however if we decide  to Shard we should always have our Shards as   a replica set and that's important because each  Shard is a component in a shot at cluster mass   kept high availability the Shard is are immutable  and the Mach size of the Shard is 512 bytes so we   can now Shard a collection on any key that is  longer than 512 bytes the Shard keys must exist   in every document end up in our collection and  if we choose range based Sharding our data may   not be distributed evenly across all the shots but  if we choose a hashed based sharding our data will   be distributed randomly and this is important  because when we do a query we may have to run   our query across multiple Shards instead of  just pointing our query to a single Shard thank you any questions compound Sharding key is shutting our  collection on multiple keys so for example   if we have collection users and we want to Shard  on user ID the compound key will be short on user   ID and created at for example if we have some  date and we should probably do that if we have   if most of our queries are running like select  client user ID were created at greater than or   less than or we should aim to create our short  key based of what our query patterns look like   if all of the query patterns that we have in  our application start with the Schottky pole   that's the best short key that we can find the  ideal Shard sorry the so the question is to do   the Schottky have to be indexed yes the short key  itself must be indexed or it must be the leftmost   part of an index we can Shard for example on  user ID created that but the index itself can   be user ID created status and some other keys  yes initially when we have the collection and   is not Sharded we know that our key is unique  or is not unique and if we know that our key is   unique and we want to Shard the collection  on that key we can later when we shut the   collection we cannot add any more unique keys  in the collection and even if the Shard key is   not unique and we decide to shut the collection  later on we cannot add unity in mechanics again we're running out of time any other  questions so you can meet me in the   booth we have pity on booth just drop  by and we can discuss more thank you

Channel: Percona

Views: 33,895

Rating: 4.7710843 out of 5

Keywords: MongoDB, database tutorial, mongodb tutorial, percona tutorial, percona, optimize mongodb, mongodb fast, open source clustering, data redundancy database, sharded database, sharded mongo, what is sharding in mongodb, what is sharding, sharding database, mongo tutorial, mongodb database, mongodb sharding, sharding in mongodb, mongodb sharding and replication, mongodb replica set, mongodb replication and sharding, mongodb sharding tutorial

Id: oH-gQ4JdXQc

Length: 49min 43sec (2983 seconds)

Published: Wed Nov 15 2017