PostgreSQL + ZFS: Best Practices and Standard Procedures

we're going to do a crash course lightening talk on CFS and Postgres and most of these things are applicable to not just Postgres most of them talk about is filesystem related and storage related and how it relates to databases but you should be using Postgres even though we're at a MySQL conference so if you're not using GFS you're losing data this is pretty much the most important thing that you can take away from this talk is if you're not using this you're losing it and you don't know it you probably are so Clark Street laws any sufficiently advanced technology indistinguishable from magic ZFS is magic but it is it's not quite magic but it's impressive software regardless so there's many things that go into ZFS and if you follow political you know humor maybe that's too soon maybe it's not so in the ZFS world most everything that you do is sorry most everything that you do you know you're used to tuning things in ZFS wide you kind of do a handful of things that work with regards to tuning but generally speaking there's not a lot that you need to do so we're going to go through a handful of things here and we're going to move through this deck pretty quickly largely because this talk was originally oriented towards a 50 minute talk and the talk got shortened because of scheduling down to 50 minute our to 25 minutes so we're going to move fast so one of the most important things about database is the fact that its stores data on disk so what does it mean to talk about what a database does with regards to its storage of bits on disk because that's really what we care about that's why we use databases all of your stateful applications end up on disk you know as translated through an SQL layer MySQL PostgreSQL whatever and at the end of the you know you've got some backing file system or storage organization of storage and we use ZFS or I am recommending so when on the storage side of things when you see a ia right ahead request come through you're going to have the database system issue a right for the right ahead log of bin log whatever and that's your intent log and it's going to show up as an 8k age fantastic it's a P right it shows up is basically sequential i/o this is very compressible sequential i/o but it also gets turned and turned over pretty quickly pretty simple workload this is basically a sequential append operation very nice and performs very well in CFS because ZFS we're going to get to is a log structured file system the most important part of databases on ZFS may have to do with data integrity and the ability to move forward and backward in time the ability to take snapshots in basically a pause this way taking snapshots on ZFS takes handful of microseconds compared to a normal block storage file system where it actually has to freeze the entire world and walk all of the available I nodes which is a terrible place to be if you're trying to be a 24 by 7 operation you don't have an off duty cycle that you can potentially take a backup theory you should be able to take these kind of backups and and do this maybe once a minute and that should be an OK thing and in fact it is when you're using ZFS and it's not the case with any other filesystem so use the fs backups or a fantastic thing it's very simple backup or a start PG start backup to do the logical flush and you take your snapshot and say stop backup so that's fine and all but what you also need to do is you need to go and flush this data to disk and so what you actually need to and credit a memory coherent snapshot is to take us checksum and flush the data all your dirty buffers do a start back up grab some data or however you're going to do it and stop backup in this particular case this little ray in the backup utility happens to be ZFS now and at that point in time you actually maybe not don't even you can can simplify things further and remove that checkpoint and you know just use your three commits and so you've got a very performant fast and you know reliable backup system that comes in the form from ZFS for regardless of your database but in this case Postgres you know in basically three steps and there's not much as you can't optimize this problem any further you're always going to have this three stat this set of three steps so great at the end of the day you know you you figured out how to do efficient Postgres backups so ZFS and Postgres are both databases one of them is database of Records one of them is the other one is a database of blocks on disk and there's lots and lots and lots of parallels between ZFS and Postgres okay so Postgres is very much copy-on-write you've got as a result and in Postgres T now you've got blocks that are checksum CFS everything's checked summed the ability to to you know perform sophisticated maintenance of your of your of your information and compress things in a way that is abstract abhi cuz you've got good separation of concerns these are all kind of first-class citizens of using when you're using ZFS or some form of log structured file system ZFS really being the only you know option that you have available to you so let's see you know there's lots of different ways that that you know when you're talking about separation of concerns and the ability to have different checksums or different layers and different layers handling different things on the checksum side of things you have the ability to you know abstract from the database the ability the the notion that that a particular disk was returning some bad data so you know how do you end up with bad data in a database there's lots of different ways of dealing with it of having this problem manifest itself Perrier's are one of my favorite firmware bugs all these are hidden away from you when you're in the when you're using some form of file system that has built-in integrity ZFS has that you ZFS okay so very fast high-level overview of what ZFS does and why it's kind of useful but the internals are really important thing and understanding the internals lets you use the technology for what it is and incorporate the value-add and features set that comes from ZFS into the rest of your application designer or database design so if you've got a normal block set filesystem which is basically an entire Linux you have a when you issue a write you write to a file descriptor descriptor and you take a buffer and you D stage that down to disk and you and you make this this VFS call and you pass that buffer from usual and to kernel and the VFS layer goes and does look up and figure out where that file and that buffer exists on disk you write that buffer to disk that's in some offset in a file internally the kernel then maps that buffer to a handful of system buffers and then under the hood it oh sure that's water um under the hood Poyser fantastic sorry under the hood it's actually taking that buffer and effectively doing like a scatter gather operation and writing that down to different logical buffers on disk on disk these logical buffers then are scattered throughout the platter and it's the operating system and the file system that actually are responsible for going doing the fetching of all of these blocks on disk and you know in assuming that the little circle there is actually spinning rust you know and these are your blocks you know you can see that when you go into a sequential read of data off disk because you're trying to read some some random file in this case you know something out of the global catalog it's internally doing a bunch of random read operations off disk but in what you thought as a user was that these were actually like logically ordered blocks and that's not the case come so you know if you have a right count to 8k buffer what that actually means then is that you know in detail here is you what we want is we want aligned boundaries so we have an 8k buffer and it goes and chops that 8k buffer up into logical blocks these are normally 4k doesn't have to be but typically for okay let's say they are at least that's kind of the modern standard at this point time and internally goes and does exactly what we had earlier where it goes and set fan out and writes that data on disk okay fine this is block file systems 101 this is what's happening when you go from you know PG data global one to some physical storage layer this is high enough level but gives you just the taste of the detailer so when you go to update an individual record or a bit any bit in a row of in any kind of a database what actually happens so that tuple inside of a table goes and results in rewrite of the entire buffer and that that so one bit here if you go and make change enabled from from true to false let's say you have to go and make an 8k right and that's what we call right amplification so you know the one of the recommendations that come as a result of using Postgres on ZFS is you can turn off full page writes because under the hood you're making an 8k write logically but you don't actually be Postgres to make that entire 8k write again because the copy-on-write semantics actually are happening under the hood inside of ZFS and you can reduce the amount and the size of this write closest call here so this is an important one from a performance or write performance perspective so at the end of the day then you know when you make your a kay right and you want to chop that and and and scatter gather that information down to logical blocks you also care about the order ZFS provides this ordering guarantee which means that if you go and have a checksum or checkpoint and you need to replay all of your logs you can rely on the fact that when your application in this case Postgres transits from user to kernel that the zi OS that are the i/o operations that happened to ZFS will come back in order when it goes in does its checkpoint replay so that works out really well because now you can guarantee up to two file system transaction semantics boundaries that IO operations have happened and they've happened in a particular order and that's really important for checkpoint and replay because you know the order in which things happen because going by to the wall log from earlier the intent log you've already written out a stream events in sequence and you can walk that and and get this good data guarantee in order to get back to a consistent place which is the long way of saying that on ZFS you don't have to worry about doing data and database integrity checks because you know in the event of a catastrophic failure your database is going to come back up you're going to do your checkpoint replay and your get back to you know a transaction group boundary whenever that checkpoint restarts or is finished so on the ZFS side of things now you can see that AV dev so physical side of things right you've one of the things that ZFS does is it has decoupled the logical representation which is to say a file is decoupled from the actual physical storage there's no the block management where bits are actually stored it is hidden through hidden by this this is this device abstraction layer okay great which allows us to take mount points VFS mount points and abstract them behind data set attributes inside of ZFS so I'm sorry data sets so data sets here have independent attributes these independent attributes you can then mount onto different directories and this is important from a database perspective because different directories here you're going to store different files or different types of files and those storage characteristics of those different directories are logically grouped that logical grouping allows you to tune these parameters differently so what we see then in ZFS because ZFS is this very large five hundred thousand lines of code it's enormous complex piece of software is that IO operations come in from the top they go through the POSIX layer they map into this this database the nosy DB the ZFS database they map through to actual a Dan allocation system and scheme that go does dispatches to the individual disks this goes back to the original slide that I are one of the original slides there where I said that IO operation or that ZFS and Postgres are reasonably similar because they're both copy-on-write databases effectively so what does this actually look like and I apologize I am trying to move through this to get to some of the best practices here at the end but some of this hopefully will resonate on some level is that if you have back to that update if you want to go and twiddle a bit on disk internally ZFS goes and builds this out as a Merkel tree and so if you have a block that you want to go and make a change on you the change that you make is actually down here on a leaf node and internally goes and rewrites this block chain all the way back to the uber block and at which point things are changing if and if this is the way that thing you know you're walking a b-tree if you do this in a normal block filesystem at what point do things become inconsistent in C I'm sorry in a log structured file system at what point do things become inconsistent and they don't right I have a hidden slide in here sorry it doesn't ever become inconsistent because you know the the visibility of the blocks at t1 were never visible because you actually Traverse from top down in this Merkel tree starting at t4 so ZFS as copy-on-write allows you to do a bunch of interesting things from a performance perspective so what actually happens is you make this twiddle you update a copy of this particular page and you walk this change all the way up to the top of the uber block and at no point in time if because you have consistency at t1 t2 t3 t4 at no point in time were you in a position where your file system was inconsistent there is no point at which a ZFS file system can be in any consistency as a result there is no fsck right so if you have very large data sets you never need to file system check the system bounces let's say hardware data center power loss kernel crash doesn't matter that when the operating system comes back up it can go and get a consistent view of the world this is from a you know a data structure internals perspective this particular point is really important and it allows you to go and leverage some of these kind of like semantics in a way that lets you build an application that's potentially faster than you would normally be able to if you're on a block start a block file system so again the answer is never never inconsistent so actual tips take tricks and tips so compression compression ends up working out really really well for Postgres and mysql even so both databases deployed on the petabyte scale both lz4 I'm not even both how you do lz4 is what you should use for all databases and I will make that statement as a carte blanche recommendation for either MySQL or Postgres for SSDs or spending rest it doesn't matter for bin logs and wall data doesn't matter use compression it ends up being a net win we replaced fusion-io with SSDs using lz4 and we outperformed fusion-io to two what two point eight to one is not an uncommon compression ratio for Postgres arm across petabytes worth of data there are cases where this was very good a hundred to one and at times we did have and the background here being a large engineering organization with a bunch of micro services we had some database instances where they were storing binary data and it was encrypted binary data which meant that the compression ratio was garbage but generally speaking this was good good yes and the reason I do is because lz4 has a short circuit on its compression so if it notices that as it's compressing things it's not getting anywhere it will short-circuit the compression so if you were using LZ for then you do need to manually enable or disable LC for I'm sorry LZ JB you need to manually the yeah that's exactly if you're using lzj be incorrect in compression you do need to manually enable or disable LZ for automatically does set for you Z standard whenever that shows up you would automatically do this for you as well yes and really like it doesn't matter what your media is always use some form of compression ask me later about why maybe you shouldn't use gzip but in all other cases basically rely on the LZ for you can go in query this this is not uncommon in practice don't mix and match LZ for in gzip on the same box and because the latency characteristics of the i/o is going in and out of the system changes for the worse when you enable gzip 9 you do get better compression out of this but the kernel wait time for all processes goes through the roof for reasons that actually have not been entirely figured out today to the best of my knowledge so this is an interesting one if when you go and write things and you're doing this log structure to pen because if you go back to where that tree diagram was all of your rights actually end up showing up as append operations and which means that all rights that happen in the same time boundary end up having locality on disk which means if you go and read some you know related attribute it's going to get read off disk at the same time because it's on this potentially the Senate it's a very probable that it's written on the same for K page of death data because they all of those i/o operations ended up on the same page right on a block filesystem you may take some nibble and put it off on like block 1 and some other nibble and though it's going to want block 2 you know you're going to have to do a random seek especially on spending medium in ZFS those may actually come back and end up on the same physical page so this ends up being an interesting performance win if you go and compare that with PG repack which is the equivalent of like you know doing a I'm going to call this call it a defrag just to be technology agnostic but like to do like you know some kind of a repack it you can actually go and get very very both good impression and performance because you have a high degree of locality for related data 'sm slide where is online later so you can go and get those links then I'm more worried about time than anything else which is again why moving really fast so yet don't ever use D dupe and I know that it's a feature flag don't ever ever ever do it there's lots of reasons not to do so the reason I came back to slide one there was when I made the comment if you're not using the FS you're losing data this has been studied on reasonably large scale this was an interesting study done of both a by CMU but at Facebook of their data set which is enormous in larger data set than what I was working with at the time very skies his numbers with regards to uncorrectable errors is in line with what we experienced on a smaller scale or smaller scale was on the order of magnitude petabytes and we were saying single to double-digit rates of uncorrectable errors on SSD media without ZFS that would have just sent the data would have silently corrupted without our ability to detect right it's not that the errors don't exist on other file systems it's that you have no way of knowing with ZFS you can actually detect that and not all SSD vendors are created equal this particular vendor I can name I know exactly who that is because their error rate was off the charts awful oh yeah amazingly bad so huh this is being recorded asked me afterwards so you know 4.2 percent to 34 percent this is common this is industry common right you probably want to be closer to single-digit but having double-digit is actually not uncommon so if you think about this uncorrectable bits per year do the math on this and all of a sudden you end up in a situation where you're ending up with you know 99.9 percent of your servers have at least one piece of bit rot per year all right you put enough disks in there and that's kind of scary right and that's on metal right on cloud providers actually abstract some of this from you with EBS where they actually do their own eraser coating on this not erase your coding like s3 brace your coding but they do their own error correction and you don't have to deal with this directly I have seen people send me ZFS scrubs where on Amazon they're actually seeing nonzero error rates coming out of EPs kind of oh that's spooky so things that typically cause uncorrectable bits temperature bus you know who knows what actually contributes to to to this kind of failure I am going to move faster this is what we care about is the fact that you can take a do a scrub on ZFS and it will tell you how much data has been repaired because as its walking all of its checksums and and verifying its integrity it will tell you how much data and number of errors and how long it took in order for something come back in a failed state this particular SSD study has been replicated on spinning me and pretty much every other form of storage media out there and why did I tell you that so I can tell you this I'm going to skip all of the interesting terminology and get to some more interesting bits here about ZFS specifically which is how you use it ZFS crazy pool and you're able to do this this directory manipulation and creation of datasets and directories online on-the-fly to see full lists it's very easy very human friendly very machine scriptable this is an important element of command-line tools anybody that's design command-line utilities please take note of specific flags and allow it to be easy to parse input and output uh you can set your quotas you can perform a bunch of interesting things tuning tips disable a time enable compression tune your record size and consider tweaking the primary cash because databases already have their own user land cache you don't need to have the file system caching data that you're already going to cache in user space you don't need to cache it twice if you're going to eat the data set that in when I say data set I mean per directory the ability to consume that the tune the caching characteristics of a directory on a per directory basis turn off change the primary cache to metadata only it's very easy to go do a ZFS get you can figure out what the existing parameters are make your tweaks go and figure out what it is it's all done online it takes a couple of seconds it's very easy to do we're not going to get into discussion element to this but you can change your record size and set it to 16 K if you need to which is interesting because that's a multiple of post-crisis record size why wouldn't you want to have that aligned well it's because you've got this 2.8 to 1 compression ratio so I can get 2.8 one additional you know arm chunks of data into the same 8k page size on disk this is interesting in that you can go in and basically prefetch data as a result of using compression with a larger record size you have to chase that all the way through to the physical layer to figure out whether or not that's good or not for your application because it's log structured it's also very fast you can do huge amounts of throughput quite easily where is my there we go right 228,000 right operations a second that's the number I was looking for and this was all done on spinning rust right this one was sixteen drives right so this is not some enormous sand but it is like it's it's a sizable server ZFS got your back do your scrubs make sure that that's automated it will tell you what to look for this zero zero zero you're in a probably a healthy place ZFS again tells you this other file systems will definitely not attempt attempt Donna checksums this is something that's confusing when I say this I'm talking about the amount of storage used per dataset I'm not going to get into why that's confusing but be aware of that way if you're coming into this as somebody that who hasn't used in the past backups this is great I love doing this at live I this is slide where but you can go do this and it turns out that what would normally be a catastrophic operation or MRF and Postgres is behavior this when it detects that it's entire data set disappeared is actually not that hard to recover from because you just do a rollback and that's really that simple in order to move forwards and backwards in time and doing this on gigabytes to terabytes worth of data it can take anywhere from seconds to tens of seconds that's it and that's really important because you all pretty much never want to be in a position where you're going to point in time recovery in order to restore I would much prefer this if you've ever had to go through point time recovery procedures that's awful compared to just running this and restarting Postgres cloning is very convenient where you can go in and take your snapshot and mount it in a different place and start up your database and test PG upgrade and naming conventions of your data sets which map to individual directories have some convention there don't do that entirely willy nilly it works in your favor you can move data sets between servers this is really convenient as well you can do this incrementally or whole data sets cart launch very very useful for moving data around you can take your send or take a snapshot you can send it you can send this over a pipe which means that you can send a data set or a directory of data between two different files or between two different servers because you did a ZFS send pipe SSH on the other end you do a ZFS receive you use route these are all route operations no no this is basically a trusted stream of data ah that would be nice to have I don't think that's happening anytime soon in CFS I need to talk to you hold on a second you're not allowed to go anywhere so then you can do see snapshots incrementally it's very useful and I'm not going to explain these two slides because I heard that other room clap last one log bias throughput you probably want that to be your common case bypass the Zil you don't need the intent log for your common case operations maybe for things where you actually do care about data loss up to the transaction group size which defaults to five seconds you can tune that down to one second change your log bias to throughput MySQL has the ability to sync once a second turns out if you go and do - Kuehl syncs once a second you probably are also very okay with having your log bias equals throughput and setting your ZFS transaction group sync or sync timeout to also one second I'm okay with typically in most cases with one second of most cases I don't want somebody to take quote me out of context but most cases once that kind of data loss if I get a 10 or 100x increase in my write volume okay I get substantially higher yield out of my database as a result how do you go and configure some of these things again for the slide where for later on so anybody cares synchronous commits similar in many ways but again log bias throughput nice and we are hiring I am one minute over questions you're not allowed to leave them Robert James yes yes we did benchmark this this gentleman over here Peter was involved in that we end up seeing that transaction group of one second outperformed five seconds for re and I think that had to do with the number of pages that were in the transaction group that were in the process of being coalesced to disk which meant that you had fewer number of i/o operations that were potentially pinned behind a lock because the transaction was deserializing um no because we assume that Merkle trees worked James that was on FreeBSD yeah this was most that all this demo where was done on FreeBSD yeah and it also works on a Loomis I here Oh which doesn't work that's true yes it it sorts the dot first yeah which is a clever thing yeah no this was all done on in this case bsd ZFS on linux would have the same semantics there you go CentOS as well and Ubuntu now which I'm very excited to see like these these are not BSD or Loomis specific things but they're definitely first class citizens in those environments Linux is getting their congratulations in the meantime you know there's other production-ready systems that do through this stuff very well I think we need to wind this session down and then we can talk more because that sounds like a good interesting conversation good yeah yeah yep snapshot has not gone away where things get complicated is when you're doing roll backs on clones yeah so you need to be a little careful with some of that stuff clones are fine where you pay the price the deferred cost in snapshots is in the deletes period snapshots free when you go to delete a snapshot that's where you pay the cost there's a an excellent talk by Kirk McKusick that I would highly recommend going and watching if you are interested schedule your deletes for the middle of the night because there is IO associated with that and if you don't have a middle of the night then you should probably be doing this pretty incrementally 24/7 and just accept that it's like a steady state part of your workload oh definitely this is a hundred percent of a publicly traded company is running on this very similar setup now 100% MySQL and Postgres so yeah and it's been very valuable to be able to go and run the roll backs on on on a per minute base or taking snapshots on a per minute basis and then being able to roll back whenever you need to the mean time to recovery because you have these once a minute snapshots is very very valuable when you don't control the behavior of developers that are potentially doing drop databases that's a different question I'm not trying to answer but as a backstop to the fact that developers do run drop database on a periodic basis for reasons that no DBA could explain you do need some kind of way of solving that kind of random entropy so and ZFS rollback works it take them once a minute it's cheap you do have a cost don't let them accumulate forever but like you know there's a bunch of good scripts that are associated with that Zil no log bias Ark for a second typically not not if you're you can write there's environments where that makes sense but generally speaking no if you're your primary data set is on SSD doesn't buy you anything if you're coming off a spinning rust and like you know your i/o latency is like 10 15 milliseconds let's say then yeah make sense to go and have an SSD l2 work but yeah yeah SSDs are getting cheap enough that you could just go that route depends on your size your data the compression gets you a lot even on SSD sin' no actually this oh I I totally skipped over this when I was talking about consistency and naming conventions for data sets we put the both the tree wall logs and the actual data in the same directory so that you could do a ZFS snap send and get everything back coordinating it across transaction I did across different data sets doesn't that idea doesn't fly so if you go and put an entire database including its transaction on the same data set it works just fine then you can snap send and move it the performance advantage of having your transaction logs off to the side when you're using log BIOS equals throughput it goes to zero correct yeah we found prefetch to not be conducive to performance we found compression to be a win using the same TransAm data set to be a win 8k performed worse than 16 K which was interesting because you know it turned out that in not always but in what the common case the reason was is we are basically pre faulting and prefetching adjacent pages because there's enough sequential i/o and either indexes or tables to want to pre fault and preload the next jason you know record em into memory yeah there you go yeah it it doesn't surprise me it's you know take some of these tunings with as with a grain of salt because you have to go and test it and like we had different workloads where it ended up being different but generally speaking like if you had to go and like to stamp out some defaults here's some defaults that work pretty well in practice yeah yeah I definitely do not put any of this stuff behind RAID controller cards the worst thing you can do and I will that is a great set of story so that I can get off into off-the-record is why you don't ever want to use raid cards use HBAs maybe SAS expanders if you have to but use HBAs and direct wire disks if you can expose individual disks back to ZFS so that ZFS knows what's going into your pool or tank or whatever data say whatever you call it mirrored v-dubs works best so if you have a bunch of mirrored vtf so like if you've got into 24 disks you have 12 V dev mirrors I did not get into the structure of what a Vita is that wasn't a huge chunk of the deck right I skimmed over that but you can't ever lose a Vita ev so make sure that you have enough integrity to end and safety guarantees to never lose a V dev because if you do that the entire Z pool is gone and I'll go back but yeah the gist of it was is mirrors work very very well especially on BSD they had an i/o scheduler that lets you dispatch iOS to individual devices in the Vita so Vida of Vita of here so if in a mirrored case here if this particular disc is being sluggish depending on your kernel it would actually send all of your iOS to let's say to disk 6 6 because disc 5 is like in the process of timing which was good from an availability perspective right raid Z that's fine it works just fine and actually performs quite well much better than you would typically expect yeah yeah we actually saw very we saw we saw a performance impact but it's like on the order of like percentage not like orders of magnitude so again log bias throughput hmm it kind of goes away because you're not waiting for like synchronous operations to let's say you know n number of disks the trade-off this is a little fuzzy five disks is about kind of threshold where you move from raid are from raid-z to raid z2 so like between let's say six discs and eleven disks going to subtract the parity disks find some power of two multiple and then you want to make that so like if you wanted to have eight disks and you wanted to raid eight disks eight disks plus two means that you had a veto of size of ten disks because your actual data size was eight disks right which is a power to raid z3 at sixteen disks which means that it was sixteen plus three so 19 was the total size of disks that you'd put in your v-dub yep yes yes that's very much matches so we used anywhere from 8 to 12 v devs that were mirrored mirrored yeah but don't lose both disks that's you start to run thin there or can so if you have if you know that it's going to take a long time for your operations team to come through data center operations and replace a particular disk maybe you want three disks in a mirror Wow hard cap on the ark we can talk more about that later because I'm not sure who's coming next and this is great I like the questions I'm just saying that I'm not sure when somebody's going to come in and say we're done em I know let's keep going so hard cap on the ark arks great but in database workloads you want like a a floor I'm sorry a ceiling for the Ark and then give everything else to your application yeah so in that case it was like think we kept this at 64 32 or 64 gigs depending on the workload we did want some fall through to Ark and then on the application single tenant applications we give something like 25 percent to the operating system for its metadata and whatever else - the 32 gigs or 64 gigs - the ark cache and then the remainder of that - some fudge factor would go to shared buffers in the case of hosts grass or energy to be a buffer in ODB buffer size something like buffer pool area I forget I was just looking at the tunable today escaping but yeah okay then I will not go into that but we did and it worked over six months we beat on it pretty aggressively we had some pre-production stuff where we tickled a bunch of firmware bugs and had some awful that we had to go and navigate through and as a result of navigating through that awful we felt pretty comfortable now I'm no I'm no longer at the company where I did a lot of this at a very very large scale I'm now at a different company that does this on a very large scale so I can't talk about what I put in production three years ago but I know that it is still in production today and it is covering 100% of the system and a single intern is responsible for multiple petabytes worth of database now because the cost of administration is so low which is amazing and I'm okay saying that No they've already taken a company my slides these will be online fantastic thank you I'm sorry it was very hurried I wanted to get to kind of the meat of things but I wanted to give the quick primer so

Channel: Percona

Views: 2,514

Keywords: data in the cloud, replication, PostgreSQL, operations

Id: T_1Zo4m4v_M

Length: 43min 0sec (2580 seconds)

Published: Wed Nov 15 2017