PostgresOpen 2019 Evolving Towards Postgres

good morning my name is David Steele and this is evolving towards Postgres I'm the principal architect of country data I've been actively developing with post growth since 1999 and the primary auto Petri audit or PG backrest co-author RPG audit and a post cost contributor now and we're about to learn how I got there actually a post goes contributor this talk is partly about that so first we should talk about what is PG backrest this is the primary project that I work on this talk is not about PG Bracco's particularly as in in terms of the project its features etc this talk is really about PG backrest relationship with post cus but for those of you who are not familiar PG backrest is backup software for Postgres this sentence is our tagline right we're trying to keep things easy scalable simple etc etc but essentially at the end of the day it's its backup software external to Postgres but obviously that works very closely with Postgres so backrest is part of what we call the Postgres ecosystem and these are the the ecosystem is products and services that work with PostgreSQL but are not included in PostgreSQL some examples are PG admin right really common graphical user interface currently produced by e DB there's a there's a nice new fresh version of it out lots of people use it you know if you're used to desktop type operations then PG admin is your friend for connecting to post grosse post kiss which is easily the premier extension for post Coase so this is a project outside of Postgres that provides GIS functionality and and I would say this may be the one one of the few external products to Postgres where people get Postgres because of post just not the other way around so they'll say we need post kiss oh and we'll get post gust to go with that please but it's probably one of the few things that's like that but it but it is like that another example is check Postgres so this is a monitoring plug-in for nagios which is commonly you can also use with I change' and other others PG part man before the days of declarative partition part man was what you used if you were gonna do partitioning and I think Keith thought and hoped that he's not here but uh I think he thought and hoped that when declared auditioning was in Postgres that this extension would go away but it turns out that is not true it still has a life as partition management automatic creation of partitions replicating partitions to standbys so that logical replication continues to work after a partition is created etc etc so sorry Keith PG bouncer another great example this is a connection polar for Postgres that allows you to share a relatively small number of connections among a very large number of clients and of course PG backrest backup and restore there are a number of other and and it most of these categories except for post chests post kisses just post just but there are several different products you can get in each of these categories and PG backrest is just one of the available backup solutions there's also bar man there's wall g PG pro backup pitter e omni pitter another one i forget the name though there's there's a wide selection so yeah many many more so one of the questions that people who are new to the community ask is why yeah why do you have all this why is it not built into Postgres and it's it really has to do with the nature there's a couple reasons I think but one is the nature of the open source community so there are lots of different approaches to a problem so you're going to have multiple approaches right no one's ever gonna really agree completely on anything and there's this thing called Conway's law which basically talks about how software is developed based on the organization that develops it and so in this case because the community is free to pursue their own ideas you're going to have multiple solutions you know there's no strict governing body that says there will be one backups solution for Postgres even even in Postgres even if you look at core post Coast there are multiple backup solutions built into Postgres itself right so there's several different ways to do backups based on the type of backup that you need and what your requirements are etc so even Postgres itself can't necessarily decide on a single backup solution there is no one process fits all or in one size fits all but you know basically over time the best ideas can be integrated into Postgres core and this is this is a pretty slow process though you know getting external projects to features into core is pretty slow I have some examples of having done that in this talk so it's not impossible but it is it is a very slow process but essentially there's there's another reason here I think too is that it's it's a man person power issue shall we say it the the the Postgres project has a certain amount of resources and they're gonna dedicate those in ways that they they see most fit and other people may dedicate their resources in different ways so you have these external projects you have the core projects and a lot of us based on who has time and who has interest and the people who have the time in the interest or come back up may not necessarily be directly involved in the community which is how I got started on this whole thing so I was at a company about six years ago we were working with very large databases so we had about 60 terabytes of databases total the largest single cluster was 10 terabytes so we had a lot of data to back up and we were struggling to get that done and the toll we were using at the time was bar man which has made many improvements over the years but six years ago is simply not up to the task so Steven and I thought well hey we can do better as many people have thought and PG backrests was born now this is a talk about interfacing with the community talking and working with the community so but for a lot of projects I think phase one is this phase ignoring Postgres right so initially that's what I did I was working on his backup software off to the side I didn't really talk to the community about it there maybe some people in the communities are aware I wasn't involved to the extent I am now and a lot of projects go through their entire lifecycle in this phase ignoring Postgres not ignoring post Gus I mean obviously if there are post cast project they're gonna be interested in new versions coming out they're going to but I think in this sense this slide would probably be better titled in ignoring the PostgreSQL community might be a better way to to title this slide but and and you know so PG backrest did this for several years we had no real involvement with the community we weren't thinking about backup in terms of how that interacted with Postgres how we could make that better etc and and as I said many projects will do this indefinitely they never get involved the community they've got everything they need everyone's happy and there's nothing wrong with that you know I've been using Postgres since 1999 I only got involved the community about five years ago before that I was a practitioner I used it it worked no complaints right so which brings us to our next phase contributing to Postgres so the question is why did we start contributing in Postgres and it really started with with one patch in particular certainly in relation I had some other interactions of Postgres but when it comes to PG backrests interaction you know as a project interaction with Postgres it starts with dispatch so I saw this patch and this was a patch by Magnus hack endure who's here and if this was do you implement API functions for non-exclusive backup now this is the method that PG based backup has always used so you can run to PG based backups at the same time they don't write a backup label into the day directory etc etc I don't want to go into the technical details of why non-exclusive backups are better but let's just say was take it as read that they are so just for now so when I saw this patch I thought I really want to get behind this patch and make sure this thing gets committed because this would be extremely valuable for PG backgrounds to be able to use the same backup method that PG based backup uses previously it was completely internal and it couldn't be used second quadrant wrote an extension which exposed the internal API but it had some problems so I was like okay well I haven't really contributed anything anything to Postgres before but I can certainly review a patch right anyone can review you just sign up take a look at the patch so I got in I tested it I found some issues Magnus fixed them we went through a little back and forth and voila non-exclusive backup was committed I do have a little bit down here on the bottom about why non-exclusive is better but the better thing to do is actually go read the documentation the continuous archiving and recovery documentation will explain in detail why the non-exclusive backup is the preferred method and exclusive is deprecated so I thought this was pretty cool oh you know I hope get this feature committed into Postgres which allowed me to make backups better for the users now from 9.6 on back rest will only use non-exclusive mode for a backup on that version you don't have an option you can't say I want to use exclusive we don't allow it if you're 9 6 and above you're always gonna be unknown exclusive there are no downsides to it there are only upsides there's no advantage to the other method in any way so I went on for a while flush with my success of getting helping to get non-exclusive backups committed into Postgres and and then I started working on the PG backrest backup from standby feature which doesn't really work the same way that the post goes back up from standby works we use the master and the standby together the primary and standby together to generate a backup so most of the data is copied from the standby but a few files are copied from the primary and what that gives you it gives you a backup that looks like it was taken from the primary unlike a PG based backup standby backup which really does look like it was taken from the standby and there are some compromises there and they're not ideal so I started doing a lot of research into what makes Postgres consistent so the question was what files can we copy from the standby and what files must be copied from the primary and while I was doing this I ran into a lot of files in the Postgres cluster that actually as far as I could see we're completely Cerrie based on my outer to the code and it's a lot more directories than you might a guess like you know PG sub trans for instance is one of those directories that you don't need it etc etc so most of these directories are actually rien ated at restart time so I made a list of these directories and I really wanted to exclude these directories from the backup entirely because they looked useless to me but it was a pretty scary thing to do just to say oh I'm just gonna exclude this stuff so I decided instead to write a patch for PG based backup to exclude these directories and you know submit it to Postgres as for consideration because I figured oh if I'm wrong people were going to tell me right quick so the initial reaction to the patch was a skeptical but what happened was the people who actually wrote these various subsections started chiming in and saying oh yeah we just leave that data around for debugging purposes it's not you don't need it etc etc so like all the people who had actually originally written these various modules chimed in and said no they're not necessary so they got committed into Postgres so now we have these exclusions in PG based backup soon after that I found another foul PG internal dot an it which is also recreated at startup time and can be safely it scooted from backups so this was a pretty cool thing because basically I was able to use the post crest community as a way to vet and peer review what is essentially a PG back rest patch right this is something I wanted to add to our software but I was a little too scared to do it and by contributing to PG based backup I was able to get this peer review that I really wanted and if you had just in in I could I can tell you with the community if you just write any like if I just wrote an email and said I'm interested in scooting these directories from PG back rest there's a good chance I would get crickets from that right some people might respond but generally speaking that would be considered pretty off-topic right but by Framing it is a PG based backup patch it got attention and the advantage here too is that PG based backup is now a better piece of software as well so so both projects have benefited from this after I got my vetted patch for PG back rows PG based back up got the same patch later we also started we were at patches to exclude unlock tables from backups and temp tables as well because these are both completely useless they will be you know temp tables are just deleted by Proust ghosts that restart unlock tables are truncated and the unlock tables in particular cause confusion for people because some people had database with a very large amounts of data in unlogged tables so let's say you do a backup you've got a terabyte database 500 gig is and unlock tables so your backup is 1 terabyte you restore the backup your restore is 1 terabyte but as soon as you start Postgres it deletes 500 gig worth of data and so then your cluster is 500 gigs so people are like hey what happened to my data and the first couple times this happened it was confusing but after a while we we always knew if a large piece of data went away we'd say how many on log tables do you have and they're like oh yeah we have this for the yeah ok well there you go but this eliminate not only is eliminate the confusion of wondering why am i cluster size is different than my backup size but it obviously it was just a waste of data to copy this stuff like why do it so on log temp tables were actually very straightforward on long tables are a little complicated because you have to keep part of the unlock table around so there's an it file that has to be recopy into the main fork and we'll get too much technical on that but let's just say that you have to do a little file manipulation to sort of zero out you know to truncate a an on log table so you know of course we were able to basically get that code from the Postgres start up right and get that working pretty easily but again getting that vetted through the community there was also some skepticism on this like whether this would work or not you know we were able to get past that and get it committed so really like I said these were things that I wanted to do for PG backrest by contributing than the Postgres I was able to get free review and and Postgres benefited as well so the next thing so a little time went by and the next thing we worked on was a adding a new feature to Postgres right so these backup exclusions are neat but they're not really a feature it's an enhancement to a previous feature so another thing that we wanted to be able to do for security reasons is take a backup of Postgres without being Postgres user right so we want to be able to use an unprivileged user to take the backup a couple reasons for this the main reason when you talk about security you can be talking about a couple of different things so clearly the the user is actually doing the backup has access to read all of the data so we say security in this sense we're not talking about hi any trying to hide that data from the user because that's not possible because this user is going to make a copy of it in this case we're just talking about making sure that that user can't do anything bad to the data right so they're not gonna be able to delete a file they're not gonna be able to go in and say I'm out of fire row and change something right so so security doesn't have to be all you know all or nothing I mean there there are grades of security and in this case you're at least protecting the data from modification and that's the goal now this turned out to be a multi-year project to get this feature in and if you contribute any reasonably large feature to post goes I think you're going to find that it's a multi-year project that's just the way it goes so you know and it was finally committed for Postgres 11 but you can see below there are several there are a couple of commits that this depended on that we made it overtime one of which I authored with Adam right well because he did the most like a good chunk of the work for that one patch and so it might have been impossible in a way to even get this patch in if it weren't for these previous commits and all of these commits were so they kind of had to be couched in a way where they were useful to the project but also drove my project forward and this is this final commit up at the top allow group access on PG data but these other commits came before that to actually lay the groundwork and refactor Postgres in such a way that that was going to be a possible thing to do while at the same time both of these commits made Postgres better for instance in post caste the mode that the file was going to be written as was hard-coded pretty much everywhere so you had zero seven zero zero zero six zero zero just hard-coded throughout so these were partly to unify like first unifies those permissions into constants and then later make them variables so that they could be modified yes James no I mean I absolutely made it clear where I was going I wasn't trying to be sneaky like just say oh well I've got this this evil plan in the future but I'm gonna trick post cross by like no I mean it was it was I was very clear about what I was trying to accomplish in the end it's just that the original patch that Adam and I wrote it was too big to be accepted because it touched every file in Postgres right I mean really the if you go look at these commits then look at the changelog it's it's it's a little frightening especially the three patches together how many files were changed in post Coast by those patches it's but a lot not everything obviously but pretty much anything having to do with IO was was touched in some way yes so the question was about you know if there are so many changes how did you yeah how did you handle the review of that the answer is carefully there's a couple things going for us here first of all post Chris has tests so if I broke something that you know made one of the tests break then we would know there was a problem you can also use the test to the modifications that I had to make the test to make to make them work is an indicator of the behavioral change so it helps you measure the behavioral change in the code by looking at how the test changed and then the rest of it is is as any patch is done in Postgres is line by line review in this case the changes were very much according to a pattern so taking hard coded changes and making them as in this generic change i wrote new functions to because most parts of the code don't care what the permission is right they can use the default permission so there's only there's only one exception in Postgres where the default permissions aren't used and so I wrote another layer of functions to that we could paste into all those other places so we could change the permissions underneath them and not have to worry about it so you're really looking at that pattern so you have to go look at each call site where I made that modification say is in the previous code and say is it okay to use the default permissions in this case or does this function need to sometimes set custom permissions does that anything does that a pretty good answer oh yeah sorry so the next the second part of the question is what do you do about active development well we do something called rebasing so the patch patches get out of date incredibly quickly and so you have to constantly keep the patch up to date so part of the patch maintenance is actually just keeping it working with the current codebase and for a patch like this that touches that many places it was a bit challenging to keep it up-to-date to be perfectly honest with you so every time we got to a new commit fest I would have to rebase the patch go through deal with any conflicts and provide a new patch the community if even if my code had not changed and that is a constant an ongoing process and you will see many messages on hackers that basically just say this doesn't this doesn't apply anymore you know please rebase your patch and that's the process we go through to do that and and it is frustrating but one thing that's helpful is to break it up into smaller pieces and this this speaks to all of your questions is you know dealing with the review process in dealing with the rebased process the smaller the patches are the easier that is so if you can break the patch up into bite-sized chunks it's a lot easier to not only get it committed because it's it's comprehensible to people it's small enough to be reviewed say in a single sitting where you don't need multiple days to work on it and it's easier to keep the patch rebase so while I was working on some of these earlier patches I would let the later patches just languish you know I'm just someplace and rebates them only when I got to the point where I wanted to resubmit them to the community for consideration so that's adding a new feature and so obviously this feature had a direct material benefit for PG backrest and our users the other benefit is that every other piece of backup software out there which there are quite a few can now use this feature as well so it's not just for us so Postgres you know working with Postgres and working with you know as we work more more closely with Postgres we started thinking about language as well you know which language you pick for a project is very very important but is usually the circumstances of when you start a project decide what the language is going to be and the circumstances of the start of this project was we needed to backup 50 terabytes of data quickly and we needed to get software that would do that quickly so while I love C C was not really a consideration for writing PG about cost originally because I needed it done quickly and and C has many strengths but speed is not really one of them well speed of writing the software is not one of C's drinks once you've got it written it's quite fast and great so peih-gee back west's are really written in pearl but we are currently migrating it to see we were very nearly at the end we were within a few months at the end of that project and boy I can't wait to be done but people ask us why you know basically we've devoted now Cynthia and I have devoted two years of our lives essentially to rewriting a piece of software that we had already written so why do we do it one of the main reasons actually was speed the Perl code the code base has a grew larger and larger got slower and slower to start so starting up back rest started to get slow and for things like backup and restore 500 milliseconds startup time it's negligible right who really cares either your backup is really really big in which case 500 milliseconds means nothing or your backup is really really small in which case 500 milliseconds means nothing so either way it means nothing but the probable area we were having problem with was for the archive commands so every time PG our Postgres wants to push a wall segment to the archive it calls the command to do that and so if your commands taking 500 milliseconds to load that means you can push to archive segments per second this is a serious problem because there are people who do hundreds per second so and they would build up big queues which they would burn off overnight and you'd have this you know the typical graph of wall buildup you know push down so one of our first goals was to get those commands faster so we rewrote those commands and see the rest of the program was still in Perl but we had that a lot faster now at the point where we had rewritten those and see we had already embarked on the migration of the entire project but we did those first because they were the most time-sensitive backup in in Perl is not as time-sensitive although the C code is faster in some key areas and one of the areas is building the initial backup manifest so if you have a database with a lot of tables the old version of PG backrests can take up to 10 minutes to build just to build the list of things that it wants to do the C version will do that in about five seconds so so there are some there are some speed benefits to you know the C code as well now now if you have a database that large you were probably gonna spend the next six hours backing it up so again the ten minute thing is not killer but still it's nice another benefit is simplified deployment so now you don't have to have the perl runtime you don't have to the various perl packages the the libraries that we use in PG backrests are very commonly installed if you've got a Linux system with anything non-trivial on it you probably have Lib SSL Lib XML and Livesey oh and and lid PQ right so you've got those libraries installed and you're ready to go so so in that sense PG backrest the C version has zero dependencies now it has a few but as I said those things are almost always going to already be there and if they're not already there they're at least relatively well-known entrusted pieces of software with the exception of OpenSSL of course but it is what it is we are planning to also support Lib NSS and some other things in the future but for right now open SSL is the it's what Postgres uses which brings us to Postgres so this is really a talk about working with post coasts right so was was the other thing we did well we we really wanted to be in C because Postgres is in C right so it so these first two items were definitely nice and we could have gotten the first one just by doing part of you know those couple of commands to C and being done but having the using the same language as Postgres has an enormous number of advantages for us and the first thing we ever did so so the what I was just talking about was you know contributing code back to post gross so this section is a little more about stealing code from Postgres and the first thing we ever grabbed from Postgres was the page checksum code so we wanted to be able to verify page check sums during the backup it's essentially free because when you're doing a backup you have that data in your hand so if you can run the page checksum code on it it it has almost zero cost so why not do it but the page checksum code is fairly complicated it's also highly optimized in C so it's it's a it's a big ass you say convert that to Perl and actually have it be fast and then on top of that have it be correct so we were like okay well let's grab this code and in this case we grabbed it and then wrote a interface layer between Perl and C so the Perl code could use the C code and basically all it was getting back was the verification is this is the data you gave me correct or not so the the data that was going back and forth between C and Perl was really easy it's just buffers and boolean z' and so the algorithm itself was safely embedded in C where we could just copy it from post Coase and be done with it we also use PG control structures and you know initially we had to code these in Perl basically reading them out piece-by-piece but there's this thing on systems called alignment which makes this really a hassle because 32-bit systems will write structures to disk differently than 64-bit systems do so in Perl we had to detect what kind of system we were on and then add little offsets in and the further you got down into the the structure the more complicated it gods so reading the first couple things out was okay and then it got harder and harder and harder and harder so as so what that meant in Perl is we only went down so far and we were able to only able to capture so much the data from PG control now and see we have a we're able to read all the versions of PG control from 8.3 through the current version and literally what we do is we have a big header file that has a lot of if deafs in it and we copy the new structures out of the new version of post Coase and then we build a file an object file for each version of Postgres and then and then we have our own interface layer with version numbers on them so we can say give me the PG control read this PG control file as if it were 93 or can you you know is this just ask the interface layer is this your PG control version can you read this or should I ask the next person and you know go through the loop another thing that's really important to us as wall structures we the wall headers do verify that the wall that you're archiving belongs with the backup that you're sending it to so reading the wall structures is also non-trivial we have all kinds of plans for this in the future for things we would like two more things we'd like to do here more we'd like to be able to read but for now it's nice that we can just easily get that data out with C the last thing here is portability so by writing and C E and using data structures and constructs and stuff from Postgres we are we're trying to ensure that we are as at least as portable as post Coast's so any platform that post Coase can be ported to PG backrest can be ported to and that's really the goal right because if Postgres doesn't run on that platform there's no reason for us to yeah why would you care so if we can have that kind of portability then then that's good luckily when I forgot to put this on the slides I should I should do this but when I started this project I started writing it in what's called c99 so this is a C standard for it's a compiler standard so C 99 has certain features c 89 has certain features c o6 has certain features but I went with C 99 because it has the most features for and is most widely available but at that time post cus was still in C 89 if you can believe it luckily partly I like to think I was partly responsible for this because I helped talk Tom into why the C 80 99 features were actually valuable and things that we should have but the Postgres project has since migrated to c 99 so the one compatibility barrier between PG backrest and Postgres has been removed which is pretty cool so this is all the good stuff right but working with the community can also be challenging right so so the these previous slides are all of our victories right yeh so we we proposed a patch we worked on it we got it reviewed we got it committed but there are other things that have not gone so smoothly and one of those is a patch I submitted last year to remove exclusive backup from Postgres as I said I want to go deeply into it but there are reasons that exclusive backups are actively dangerous one of the worst things is that if you are let's say you're running a backup the server crashes when the server restarts it will not restart because it will find the backup label file and then try to probably go get a wall file from sometime in the past and it will refuse to restart now it will hopefully give you a hint that you should delete that file so in the case of I was doing it back up the server crashed I need to delete this file that is correct but in the case where say you restore to backup and you forgot to write a recovery comm file you will get the same error because post Gus doesn't understand that there's a difference here and if you remove that file in that case Postgres will start and you will have a corrupt database simple as that and and the worst thing is I won't say anything about it right it's like okay you delete a backup label I can now start whoo we're good to go now you've got corruption completely silent and and you you you may have no idea happen until you start hitting indexes and other data structures that are inconsistent and it won't work so given how dangers this thing is and given the fact that we've had non exclusive backup available in PG base back up since 9.2 and available to all other backup software since 9.6 I thought it would be a slam dunk for Postgres 12 to remove the exclusive mode from post cos boy you want to read a really really long thread that was that's a you know Renee was talking about order sorry we're never mind this is an earlier conversation but um you want to read a really long long post ghost thread this is one of them but basically it fell into three equal groups so they're the people who said no just no because it will it will break too many existing scripts so all the people out there who are ready and in bash one thing about non-exclusive backup as it requires you to hold a connection open to post Coast during the entire backup and this is something bash does not do very well so there was resistance on the count of all the bash scripts that it would break okay then there was a third camp and these were only these were almost equal the third camp was this is a terrific idea but let's not do it yet you know a lot of people have this notion of keeping things around for five years or like you know deprecating things than holding them for five years there are lots of instances just breaking things in a single version so this is really not a thing but when people want an argument for why something shouldn't be done they like to bring up that five-year rule which isn't really one but so these people were spacely saying yes but not now let's do it sometime in the future let's do post cast thirteen approach quest fourteen and then of course there was the camp that that I was in because I submitted the patch which was yes we should do this now because it's a better method and it's been available for years to everyone if they have an upgraded yet been you know so this patch is still pending in fact on a little afraid it's been like moving from commit fest to commit fest and I'm a little afraid to actually bring it up again because I'm I feel like we're probably still gonna be in these three nearly equal groups so currently I'm thinking maybe I'm just gonna hold it for post cast 14 when I'm hoping that the yes but not now group will have increased so I was thinking I was reviewing his slides this morning I've only given this talk once before and it was at a meet-up and when I was reviewing his slides this morning I thought this is a weird thing actually because this is not something that backrest particularly needs right so this is something you know from my perspective I was doing pretty much completely altruistically for the community to say hey this is a very dangerous thing let's get rid of it so people don't have this foot gun lying around but because PG backrest only uses non exclusive mode in 9.6 and above it's really not not an issue for us so this is something that we wanted to do just for the community just to make things safer another area where who you've been working hard is improving the backup documentation the backup documentation has a lot of really overly simplistic advice one of the things that recommends is using CP for archiving which you should never ever do so if you're reading the post ghost ox and you see the example that classic one you know if CPA exists not it don't touch it it's wrong it's bad it's horrible but updating that documentation is proven to be really difficult because what do you replace it with all right so we're not going to use CP s CP is just as bad our sink is just as bad as Italy all the things you might use to copy for files are unacceptably bad for copying wall segments and the reason is as soon as you notify post ghost that you have copied the file it believes you right so it says okay I'm gonna recycle that guy and reuse it and it's gone forever so if you for instance didn't sink that to disk or you got a network error some other bad thing happened then it's gone and your backup is no good because you can't restore past the point where you lost that wall file yes you certainly the the command as written is pretty terrible so sorry the question was you can use CP safely it's just hard and the answer is yes you can use it in combination with other tools so what you would need to do is is either you know run a sink afterwards or but except if you're copying to remote file system that becomes even tougher let's say SCP or whatever or you need to write a C program that will actually go and sync that individual file for you however as we know now there are and in many kernels there are issues where other processes cannot safely sink a file that the first process opened so both of those methods are actually unsafe in kernels or than 4.11 which almost everyone is running realistically so so actually recent evidence suggests that there is no safe ways to use CP at all previous to those kernel bugs I would have said yes there is so really if you want safety you need to be syncing the file in the same process where you open the file if you don't do that the kernel will hide errors from you something goes wrong you won't find out about it it's very frustrating so but we've been working on the documentation I wrote a patch a couple of years ago to generally improve the low-level documentation and give more warnings to whatever that was fairly uncontroversial then I wrote a patch to warn more strongly about how dangerous exclusive backup mode is and then we got into another big argument about telling users to not use a feature in PG backrests that this was a bad thing now luckily Robert came along and completely rewrote my patch he still gave me credit for it but he basically rewrote it from scratch in a way that was acceptable to everybody myself peter isin trout everyone so so thank you Robert for that because I don't I don't think anyone disagreed with me I think maybe people were disagreeing with my language maybe it was too strong maybe you know so they it was more about how it was said maybe being a little more political which is not always my uh my strong suit but luckily we did get that in we still have outstanding issues with the archive command trying to figure out a way to make that safe we've thought about doing basically a model I archive command say okay if you had an archive man command named X it would have these qualities and then use that commit you know that fake command in all the examples this isn't satisfactory to everyone because it's not something that can be directly implemented and then you're kind of forced to go to use something like say peachy backrest which may not be a bad thing but it's not the kind of thing that we want to be putting in the post Coast documentation right to some extent post guys should be standing on its own so yeah so this is an ongoing thing both of these are ongoing things and you know it when I first gave this talk it was at a meet-up and you know several the people were closely tied to the post Coast community and this default into sort of an argument in a session about complaining about various things and I don't want this to be like that it's just I want to point out that you know the community is give-and-take you can't just say hey I've got this great patch I'm going to give it to you and you're going to commit it you're going to have to negotiate you're going to have to play some politics you're going have to compromise right you may not get everything you wanted initially you may only get it over years but but it's all about caution database people are cautious by nature backup people are even more cautious so this is all about caution and making sure that post Gus remains a great stable product that we can all use and depend on so it's not malicious here it's all about making sure that the users are best served and I truly believe that even on the days when I'm really really frustrated the community I believe that it's really about serving the community and is it worth it yes it's worth it because there's a you know over time as I think I've shown you here there have been many many benefits to both projects that have come out of this so the work and and the occasional frustration has absolutely been worth it in the end and so you know that's that's my upbeat finish right because the last time I did this it was a little bit devolved into argument but that's that's kind of a meetup thing I was thinking at a conference it might not be so bad right but it Amin up everyone knows each other and so that's all I got are there any questions for the questions Adam yes the the question is about the incremental backup patch this is page-level incrementals that has been proposed for PG based backup and I definitely do not want to go into that here I I will say that my personal belief is that the feature is more advanced than PG based backup currently needs I feel like there are other needs that would be better addressed in PG base backup before going to peach page level incremental switch is a pretty advanced feature but I'm not against it because you know in the it can't hurt kind of aim but I'd be happy to talk to you with that about afterwards anything else all right well then well thank you very much

Channel: Postgres Open

Views: 275

Rating: 5 out of 5

Keywords:

Id: CjGOg88_5UE

Length: 44min 14sec (2654 seconds)

Published: Thu Sep 19 2019