Postgres in Containers - Why (not)? (Ep 254)

[Music] [Music] [Music] [Music] [Music] [Music] oh [Music] w [Music] oh [Music] hello hello live hello we are live this is on this is on nal's back hey thanks for being here um we're we're excited to get to the show because we're going to be talking about postgres all day today it's it's postgres day and uh of course we all can't I think someone said in chat already Alex said if you have data you've already failed so um that in some cases you you've not done your job correctly if you're managing databases is basically what that joke is about but um let the cloud manage the database let a let nmal and his team manage your data but if not like if you have to do it yourself that's fine too if you have to even if when AWS runs your data like you still have to take care of it so we're going to get all into that we got Lucas on the show waiting in the wings but first let's get to some of the things we do here if you didn't know this is a live stream what is this this is business this is business time and we got things to talk about real quick uh we do this show and it turns into a podcast so you below all the links all the links are below some of the links are above in the video you can't click on the video links you got to go to the description to click on an actual link um in case you're new YouTube when when you're commuting to your office when you're yeah working out on your pelaton well you could stream the video too but yeah oh that's true whatever you're when when you're walking outside and enjoying the the wonderful weather wherever you are and you want to listen to wonderful voices you could uh you could listen to this podcast where do we find it Brett we find it at podcast. bre fisher.com or where you get your favorite podcast from we're in all the places we're on Apple podcast Spotify Google overcast pod class Amazon music we we're in all we're equal opportunity podcast we put it everywhere and then everyone just choose it's like 99% of everybody's on Apple podcast like I think 3% are on Spotify no one's on Google anymore because they keep changing their podcast to the new they keep changing the app stop changing the apps please Google just keep the same one we appreciate you also you can be on Discord we have uh 17,000 devops professionals and Learners all on Discord that's a wonderful group of people that I cannot keep up with this this is all the chat that's happened the last few days that I haven't read or caught up with yet so if you're in Discord come hang out with us link right there discord.gg devops um we talk about everything Cloud native everything kubernetes and Docker we even have a wonderful little memes Channel and a retro Tech Channel I'll bu this this this one is great by the way always remember to tip your server that's a good one that was a good one that's a that's a Us North America Centric just oh right that's true uh that is a us only uh we have a jobs channel uh we have a retro Tech channel so I can uh did you know they're they're re-releasing tape cassettes now like this is a thing which no cassette players I still have some cassettes by the way of things I never did turn into like live bootleg your first tape what was your first tape Brett well my first tape that I purchased because you know I was recording off the radio like the rest of us I was bootlegging off off I was pirating off the radio waves I think my very first one was the soundtrack to Ghostbusters in 18 1980 1883 19 like 83 or 84 and my theory on soundtracks was you get more music that way like you can listen to 10 different artists if you get a soundtrack versus one artist if you buy the whole record what about you uh Paul Simon graceand that oh I love that one I had that one on tape too I love graceand that's a it's a solid album there's so much happy in that uh in that album I felt like it was a very happy album Paul Simon I haven't thought about him in a minute now that we've diverged tremendously from tech let's get back on it socas yeah so we have a newsletter brett. news get on the newsletter this is fast mode uh we have devops days and Raleigh go yes so um we have devops days uh which is a global uh volunteer run organized devops oriented conferences around the world I'm one of the organizers for the devops days Raleigh North Carolina uh conference we just put up our program we've got an amazing selection of speakers including John Willis uh doing a gen Workshop we've got my friend and colleague Carlos Santana talking about platform engineering with uh kubernetes and eks and we've got so many awesome speakers so many great lightning talks so if you're in the area or you're planning if you want to uh travel out to the to the triangle in North Carolina come check it out and uh Brett just put the link in the chat so uh sign up now registration is open and and uh you can come see me in person if you uh so choose to if you miss um I just realized I just realized it's going to be on April 11th a Thursday are we doing a live stream from there I will be on stage being the MC so that might be difficult to pull off so so so we are doing a live stream from there and then I'm running on stage to get your hot take and then I'm running off the stage okay sure all right stay tuned people you might just see us on stage you never know um we go so and lastly uh we also have a loot box we have swag you can get your swag uh bring it to the conferences by the way we're both going to be at cucon Paris next month so if you're going to be at cubon make sure to find us on socials or find us in the Discord server let us know we have an entire uh in the Discord we have a whole conference Channel um somewhere and last time at cucon Chicago uh we did a little Meetup right we had a bunch of folks from from your Discord uh meet up yeah there we go people there it is yeah we had all these folks that um so come find your community um come hang out with us um come say hello we we absolutely love uh talking to everyone that that watches the show or listens or listens yeah watches or listens uh yeah it's it was a great time and I'm looking forward to the next one so let's get to it let's get to the show because I want to learn some stuff so Brett so Brett when you've got post in containers who you going to call the Ghostbusters AKA Lucas there we goit balling welcome Lucas to the show this is Lucas fiddle the founder of PG analyze which we're going to get into we're going to talk about containers a whole lot of things about databases and Cloud native but Lucas welcome thank you thanks for having me yeah I'm glad you're here and our mutual friend well actually we all know uh Laura taco and and she hooked this up with the connection to you so that we could have a show about postris because I don't I actually had to go back and look we've talked about databases a lot on this show but we've never had one sort of focused on post specifically in a container so shame on us we we're fixing it today five years in um we're we're going to do it I'm glad you're here that sounds great so um why did you decide to pick postgres as a focus for your career take us back a moment what is it about post good old times um what happened um well back in 2006 I was uh putting servers in racks so that was my first job like in a data center working with physical servers which you know these days people don't do that much anymore um and we kind of used postcript back then but I wasn't really I don't think I was involved in that point like really where it started was um we had a startup um which is kind of like Tumblr like a multimedia blogging Network called soup um you know like the what you could be cooking so we you know all fun chokes of the name um and so um the core database um of that social network was essentially postest right like it was one big post server that was storing all these you know blogs that people had and like the posts that people made um all the you know fun memes that they were posting um and so postes was you know um our dearest you know center of the universe with all the problems that come of that um and so that's I think where kind of started you know like me just as an application engineer back then being like hey you know what is going on like why is it slow all that fun stuff um and you know over the years postest seem to stick around yeah it's still here yeah it's still and it has been here since you know 27 years like it's really old crazy I think it just it seems so underrated right now like it I mean it's like thep off ranked you know database engine of the year so I wouldn't say underrated but okay fair enough but maybe in terms of the Zeitgeist like in terms of like the new hotness um people sleep on postgress and its abilities and its maturity what we're really talking about there though was 27 years of open-source databases which I imagine when it was first like on the first 0.1 release someone was like you're never going to get a DAT I mean that was the those were the years of Microsoft sequel of Oracle like those early day I mean I'm not even sure Microsoft squel was around like it might have been very early days if not pre Pre-Sequel in7 anyway that was a long last time ago like that was very long um yeah exactly and I mean in the beginning postc didn't even have SQL right like in the beginning it was just a database that had like this object relational idea which made it special but then you know um SQL kind of was an addition um like later into the project um popular yeah yeah it became like a proper database I would say um slightly ler than the 27 years wow that's some serious lore there I didn't actually know um so we're fast forward because we're not going to recount every every touching of postes I don't even remember when the first time I used it um and now you have a company focused on performance and yeah kind talk about that real quick yeah let me so PG allies is essentially a monitoring and tuning tool for postest right um and it's something I use myself when I work with postest so you know in a sense I kind of scratched my own itch I was like hey you know how do I work better with optimizing those queries with finding those missing indexes and so PG analy initially started as a site project a little bit over 10 years ago um and we've commercialized it more recently I would say over the last you know four or five years um we're now you know have a small team working with this um our bootstrapped so no no VC money it's all you know Freedom um from uh having to IPO at some point um it's we can just build a good product and you know um build the best tool that you can use for monitoring postes and for optimizing postes very cool um so for those out there if if you have those concerns I I can remember uh mid 2000s and it didn't really matter matter which database but I can remember where I kept having to jump in back when we were doing real Hardware rack and stack like uh nmal says rack and stack um where Developers and I'm I'm guessing this happened still today but developers were writing queries not really thinking about indexing or performance and the old someone told me long Year many years ago that I and I love this sort of analogy of you don't have a performance problem until you have a performance problem and that at least that that term makes sense that that saying makes sense to me but people you as they were more successful as a company writing their websites on databases or whatever I think this was specifically a company that was um managing housing HOAs and stuff like that so they had to deal with like pool tickets for going uh for pool lessons and they had all these things writing on a database and they they kept getting more and more successful and it kept getting slower and really it wasn't even that complicated it was buy faster discs make sure you got indexes for all the queries and watch your query response is that I mean discs are S I feel like nowadays like of course the rule is if you're on a spinning disc you're doing it wrong I think right so is it gotten like are is indexing still like the step one make sure you have the right indexes uh kind of I mean I would say it depends right like it it really depends on who you're talking with um sometimes you know so discs are still a problem to be clear right the problem nowadays is network storage right so people use like EBS and a cloud EVS is pretty good don't get me wrong um but it still introduces latency right and so for example AWS you know most recently introduced a new optimized re feature which ultimately is just a fancy way of saying they use a local nmve drive because that's much faster right and so storage still you have to think about it it's just not you know spinning discs anymore it's other considerations in the cloud um but from a you know end user perspective I would say you know queries are important indexes are important query plans are important right so understanding like the database you know makes a decision how to run your query and so I think where I see to most people you know struggling from application engineering perspective is you know query is slow and then they start looking at what the database is doing and then you're like okay this hash join sequential scan like aggregate no they just got confused right like and I feel like even even as somebody who actually creates tools for post I feel like we're not doing a good enough job yet at making that more approachable so even today I think yeah go ahead now go ahead um I was just gonna say like with the use case spread like the It prest's Evolution over time has now started to cover so many more use cases and cover so much more ground that um it might be hard to to keep up with the complexity of what postrest can support while also understanding how it's making those decisions so I think you were just about to go into like what are some of the paino areas that you see your customers and and uh people that are adopting postgress go into yeah and I think I mean it's it's good that you mentioned so these days right like last two years or so um you know gen AI I guess the last year really is when Gen AI has took taken off right um and Post in a sense I think has had a really interesting development there with PG Vector right so PG Vector is a project by Andrew Kane um which essentially lets you store embeddings inside postgress and kind of you know do stuff like um RGS and whatnot I'm not an ml expert not not an AI expert um but you can use it essentially to build your AI apps right um to store some of that data that you would otherwise store in a vector database inside postgress and so in a sense you know if you have this you know PG Vector using application a lot of the things are still the same right you still have queries you still have you know indexes like there's special index types that PG Vector has like if flat um and such and so you have to understand some of the specialities but then for example to debug a slow query you still use an explain statement right you still essentially get a query plan you still have for example a join with you know other parts of your data and so I think if if you have that basic skill set you know it it applies even if you're building a modern application in the AI space so what are some of those hot uh like the top three areas that that um people don't look at or should start looking at when they're when they're starting to tune the performance of their postgress queries yeah I think and we should definitely get to containers at some point yes but let's talk about tun a little bit um well maybe maybe as as it relates to running the cont running postc in a container as well right sure and I think well the interesting thing is for the most part I think it's actually kind of agnostic to where you deploy it right so I think if you're thinking from a tuning perspective I mean the one thing that I think a lot of people obsess over but I don't think it's actually that important is parameter tuning so you know in a sense you know it's important to like set the right config settings so that you know the database is like tuned optimally but you know that's kind of a you know half a the exercise and then you're done right so it's not the kind of thing that you keep doing and so I think in terms of ongoing work really the things you know if you think about what are the three most important things to know right is like I think understanding query plans right like understanding how the database makes a decision um seeing the database as just another piece of software right like it's as if you had a function call in your own software and you just looked at the source code for the function it just happens to look slightly different because it's a query plan um but really thinking about this as you know an extension of your software versus a magic box that just does things right so I think there's a lot of upscaling that we we should be doing more I think to make people comfortable with that um I think indexes you mentioned right so indexes is I think something where um for example PG analyze we created a tool called PG index advisor which is essentially trying to give you automatic recommendations on which indexes you might be missing right so imagine you have 100 queries or thousand queries in your system it just becomes so much text in a sense to look at right you look at SQL statements maybe you don't even recognize the SQL because you usually work in an orm on your application side um and so what we've created to kind of solve this problem is essentially a way to to do indexing better and I could probably you know fill a full hour talk about indexing but suffice to say you know whether you use a tool like ours or you kind of make index choices yourself I think most important there is looking at the whole workload right so not just looking at an individual query but really saying what are the different patterns how we're querying um which columns are we using which operators right so po say like these equality operators and like containment operators and then PG Vector has all its own set of operators um and so you got to understand kind of how your queries look at your data to then make the right index choices um and you ask the third thing and I'm sure I could come up with one but you know we could also well we'll probably come up with it um there's some good questions already and we might come up with it after we start talking about containers with respect to Containers let's if you should we pivot to that I remember explicitly talking about not doing anything stateful in containers a decade ago right as containers Brett right like that was you know don't do stateful um obviously that story has changed dramatically over the last five years I would say U maybe even a little bit longer but um yeah so tell us tell us more about post Cresent containers like is it is it an issue like do does it matter anymore is it are the pain points all gone and it's honky Dory no I mean I would say so compared to you know 10 years ago I would have probably told you eh you know maybe think more about why you're putting in a container I think these days it really depends I think what your context is like today it's maybe 10 years ago was probably not right so it definitely has changed I think really you know I mean if I look back 15 years ago we didn't even have Docker right we have like lxc containers and whatnot and definitely then you you know there was no point in running Post in a container and so you know for example the postcg stalker image right makes it really easy to just start a postgress container so it's not a problem to create a posters container now the question is what are you really doing by launching such a container right um and I think um like the you really need to think about the statefulness of the whole problem right so like if you're just talking raw containers not talking kubernetes right just a Docker container um ultimately you know start by thinking where your volume is at right so I mean the simplest mistake you could make I don't think people would do this but let's just say you don't do a volume Mount right so then accidentally you drop your data because you're like doing something like do compos down and then you know it just drops the volume and so I think yeah I've been there on my own machine locally so you know um not in production luckily um but so I think you know we need to think about um first of all is it production or defo staging right so I think for production I don't personally see a big benefit of using something like Docker for just running a postr container I don't think you're gaining much right um I think there are for production use cases there's a lot we could talk about on kubernetes right um but so I think the just thinking about the like I've so just over the years just to give you some context on where I'm coming from so I'm not necessarily an expert on this field but I've seen various ways of doing this right so there is I'm sure other opinions out there but to just give you a context what I've done in the past so I've um been on the um of team that was running a database as a service where we were not using containers we just had you know virtual machines that we provisioned for customers um this was back at a company called cus data where we were doing distributed postgress and so we had you know many servers many virtual machines being provisioned there was like a Central State machine that was kind of controlling you know bringing up VMS tearing them down kind of like a mini kubernetes of sorts right but like built as a custom State machine um for managing databases as a service and so that's one of the experiences I have you know actually running postgress um in production um and then later um when when I was Microsoft I um you know kind of was involved with different ways of deploying containers there and so we did have both a project to use um kind of postgres in a kubernetes environment and building a custom kind of operator for that deployment um and then also there was a project that was running postgis inside a container just inside a VM so was just essentially a way of managing the packages more efficiently um and so and many years ago also I've run posters just in Virtual machines myself right so in these days just to be clear I actually don't run postur myself um on PR for example behind the scenes we use Amazon RDS today um so we don't actually manage our own postgress um in a virtual machine or a container um but I think what I've seen over the years right is that of all the different approaches there the one that I haven't really seen a benefit of is just running posts in a container because the container like if I think of my application in the container right I gain a lot of benefits right I have you know the ability like so our application is Ruby um and rust and so just you know the ability to have the right Ruby version and not worry about you know dbm based system you know like configuring things correctly like containers pay off very quickly right so we deploy our application using containers right so that makes a lot of sense but postgress is not that way right postgress is really just you know one package maybe two packages that you really need to have like the server and contract packages um and so in that sense you're you're not you know you don't have this complexity um of managing the different um things to install and so I think you know in a production environment I don't see a benefit of just Docker and postgress um but I do think kubernetes brings additional um things yeah we have I'm thinking of like the maturity model right of uh a developer that is all on board to Containers so their first sort of fora is the standard official postgres image uh from Docker Hub um there's variance of that like there's I think there's an Alpine version probably and then there's the um we have arm and inel versions and uh then we have people in chat talking about like ha and some of the other I guess Edge case it's not really an edge case but like the things that AR important yeah it's not really in not really a part or focus of the official image um you mentioned before the show something that I I don't know about because like you said I don't I don't run my own postgres daily in a production sense so I'm not constantly worried about it if I'm going to use it it's going to be a cloud-hosted uh RDS or something like that it's going to be like that but you mentioned this Cloud native PG before the show um seems to be kubernetes specific that's right yeah and so I think what's really interesting with postest so first of all since somebody asked about a in the chat I want I do want to briefly address a and I think we'll talk more about it but one of the things you have to figure out right when you're running posters in production is how you're managing when a host goes down or when you there's some fault that requires you to switch over to a replica right and so back when we were running our own kind of system for managing virtual machines to run PR this was one of the most important things right is having a safe machine that knows hey the host has gone bad fail over or recover from backup or do all this stuff that you know you essentially expect in a life cycle of a database right and so when we look at how to run posters in kubernetes there have been many different ways to do this over the years right so the cloud native PG operator um is one of the more modern ones I would say that you know came out I think roughly two years ago um that a team at EDB is managing and it's a community project with they're kind of the main contributors and so this is not the official operator right there is no official operator but this is I would say um one of the ones you should be considering when you are thinking of running Post in cetes um and so what the team with Cloud native PG has done which I think is really interesting is they really they spent a lot of time thinking how do we like this life cycle management of a postc database right like these primaries these replicas the backups right like the archiving of the ballst stream talk more about that if you want but the point is right there's all these these kind moving Parts you have to manage that are not just the running server in they've really spent time trying to say how do we map this to kind of the The Primitives like the basic resources in kubernetes right how do we make this so this is really truly a kubernetes project right and not something um that you know feels like it belongs elsewhere and I think um for example when you look at other operators byes so for example um one that I think has had a lot of production um use also is the crunchy database um post operator and that one for example um if I was using um Post in production on today I would seriously consider this one not necessarily Cloud native P one just for its history right because this has been around for many years but you can definitely see that they architectural decisions they're not you know as as I would say absolute in in terms of you know how how to do the best possible way of mapping a postris life cycle onto kubernetes I think um this is more of a you know practical let's just get this running let's solve the you know let's have a way to do ha let's have a way to do extensions let's have a way to do upgrades um and so it's been really interesting to see these different takes on running Post in kubernetes do you have to have a solid way of handling your PVS your your uh volume mounts and uh the um yeah the PV claims and all that kind of stuff before like there's some prerequisites for using these operators I'm assuming yeah I mean you definitely need to wait to provision volumes right um I mean I think what's interesting is the um so the cloud native PG operator actually doesn't use State full sets um so to manage its state it essentially they built their own logic um and they have like arguments of why they did it and apparently one of the arguments they have is um if I capture this correctly and please you know somebody from their team listens to this please you know F like in the comment tells me what I got wrong um but my understanding is that um one of the things you want you have your database do is resize volumes right so like your data is growing right so you're giving it a certain initial allocation and one of the issues with state set seems to be um if you just you know use the standard way of how it integrates with PBS is that it it it's not easy to just resize a volume and so one of the reasons why the cloud hpg operator essentially went with their own implementation of stateful sets um is because they thought that it was better for you know if you have this online resizing where you're not actually taking down the Pod then it would be better um to have their own customer implementation in my understanding from an end user perspective that doesn't necessarily mean you need to provision things differently right I think you could still you know if your kubernetes is hooked up to a way to provision a cloud net storage right then that's essentially what you need in my understanding um but it's interesting that they found it necessary to kind of go beyond the you know Basics that kubernetes has to offer in that sense yeah The kubernetes Primitives interesting there's a whole infinite surface area of stuff to talk about with respect to that I'm sure um we got some questions you want me to throw in some questions real quick yeah we we first of all Brett we've got and Lucas we got a lot of people liking postgress in the in the chat right now lot of people yeah me too um which one should we start out with Brett so um well real quick this this uh might be a quick one Z's asking kubz is also a PG operator any opinion about it I don't have an opinion about it um I don't have any personal experience with it um I would just you know if you're looking I mean maybe an interesting thing we could talk about briefly is how should you assess an operator right like why like if you're making a decision between Cloud native PG and the country posters operator and the K operator and the St operator and all the like there's like the Sal operator I you know list goes on and on um so how should you make that decision and so I think one of the so somebody else mentioned ha I think in the chat right and so one of the things I would definitely try to understand is how is it managing ha right so if something fails what happens and so for example just again comparing the two where I have a little bit more knowledge about um which would be the crunchy one and the cloud native PG one so the crunchy one um you know kind of interacts of existing Community projects essentially um and that's actually what a lot of other operators also do is they use a project called patroni um to do failovers and Patron is essentially a like think of it as a way to to manage State again right so the problem here is who makes a decision when the database has gone bad and so what a lot of The Operators do is they use project whole patroni which the team at cando actually created initially they also made their own operator um but even if you don't use the cando operator you can still essentially see Patron used in a lot of these projects because it is a good way to say you know when when has things when have things gone bad when do we fail over and so as you're evaluating things right one of the most important things to understand is is this using Patron is this using something else Patron has a lot of production experience but it's also has a bunch of moving parts right and so what the cloud native PG operator for example has done is they actually built their own implementation of failing over nhha which you know in a sense sounds good right because so they they're describing it as we did this because we felt that it was better to use you know built in kubernetes ways of doing this in managing state so in a sense that's a good thing but it also means like if I you know look at that that I'm like you know this seems a bit fragile potentially right so because they essentially reinvented the wheel which I think is probably right choice in this context but it's still means like if I'm looking for something to run in production right now I might want to have something for ha that has you know a lot more history essentially over the gears right um and so I think I would say ha is probably one of the more important things to understand um I think somebody else um in the chat also mentioned multic clusters right and how can I manage you know post replicating across clusters and if that is one of your use cases definitely try to understand like how um you know um how the operator might be able to manage or not manage a postest that you know runs across you know different regions different azs in the same cluster right so can you have like Po Affinity right can you say place you know the replica in a different a because if one a goes down you want to have sure you know that Things fall over correctly um can you you know kind of have things replicate into a totally different cluster right is there a way to essentially get the replication stream synced up um so for example the cloud native PG operator um it does support kind of you know cluster to span stones but then if you had two different clusters that you're synchronizing it requires you to do a manual failover which is probably what you want in my experience right so if you're really going this like cross region like I have my you know us region and my Europe region and if the whole of us is down what do I want to do probably not automatically fail over right because if you for some reason have like a brief Network outage or something and then suddenly everything starts moving to Europe that's probably not the right thing um but yeah this is how I would kind of look at like I would look at these you know fundamental how does it how does this decide where to run how does it decide to fail over I think that's really one of the most important aspects so it seems like you you have to start with a a pretty good understanding of your risk profile with respect to the data and and the use case for postest then match that against the ha well guarantees or the ha implementation of these different choices and whether that's appropriate for your risk and resiliency uh Pro profile that you're trying to achieve um which is general advice for cloud native anyway yeah and I would say I mean maybe just to mention this again right like you should definitely ask yourself is it right to run a database in kubernetes right because I think the answer to this might be no right like the answer might be you should be using Amazon RDS or as database for postgress or GC Cloud SQL um like they will manage that whole state machine stuff for you right and they'll charge you a nice premium for it right so don't get me wrong if you if you're optimizing for cost it's not the right way to go necessarily um but if you don't have the expertise to think about these problems in your team it's probably better to not assume the operator magically takes care of it right because there are things to understand about how this works and so unless you you know want to get into how the operator handles these certain conditions it's probably better to just defer you know running a database to the cloud provider yeah yeah there's I'm not gonna yeah I mean AWS gets money anyway right you run your database in kubernetes you run your database in RDS AWS is happy so yeah I think we've been saying I think it's a pretty consistent message we've been putting out for some time now that uh just because you can doesn't mean you should and that is very very much the use case of databases um I mean it's it's funny that you one of the first things we we started talking about here when we're talking about all these different operators is is like volume expansion because I mean we were dealing with volume Expansion 20 years ago we were dealing with when we we started virtualizing storage and we needed to you know didn't even have containers yet we were dealing with challenges of always online databases and how do we expand those volumes and then we had um when sort of the concept of um try remember the right term we used back then uh you have sort of volumes that look bigger than they are I mean some one of you probably knows the right term for that uh like so there's so there's really only 100 Gig in the storage but your volume looks like it's a terabyte and yeah that way and we dealt with that for a while we're like well this solves the temporary problem of making it look like there's not enough storage we can add storage later and then there's actually adding the storage um I mean I've I've got so many Battle Scars I feel like from keeping databases on and one of the questions that's maybe not in line here that we're going to get to maz's question next but one of the things I I keep thinking about is if we shift from the tech for a second and we talk about the people and the archetypes or the roles that are today I haven't really worked heavily with a team that's sort of dealing with large you know terabytes and terabytes of databases in production uh multizone multi um region kind of scenarios I I'm sort I'm in the devop side nowadays so I don't have to deal with that anymore but maybe my 2000s the 2010 2015 era was more working directly with a database team and that was back in the old school days we didn't really have the term devops necessarily at the time but you had a developer team you had an Ops Team sisis admins like that's where I was and then you had a dedicated database team and I can remember very heavily working with this dedicated they worked full-time all day on Oracle and Microsoft SQL and all these sort of traditional Enterprise Solutions do you what does that look like today like in terms of the makeup I think there's a lot of people that are may be coming to this and they haven't worked with larger teams or even smaller teams that might justify a dedicated uh database administrator does do you even see that anymore is that I mean obviously like the large companies or the aws's of the world have it but do you see teams of 10 having a dedicated sequel person there or is everyone expected to know how to expand volumes and like do HA setups and SQL and know what backups and restores look like I mean there's just so much to it it's hard to imagine I mean I think Al also some of these things are definitely more on the infrastructure side right like they're not necessarily like it's the same as if you're running anything else in kubernetes for example right um or stateful things um I think what's interesting is um when you look at you know how def Ops and system ad kind of relate to each other right so like 20 years ago you were assist admin and today you definitely would not be assist admin anymore like I it's rare that I talk to anybody who identifies as a syst administrator and the same thing I think has happened with database administrators where people unless you talk like large Enterprises with like you know history maybe they're using Oracle they probably still call themselves tbas but I think for the most part these days you would probably see somebody call themselves data engineer data platform engineer um sometimes application dbas right and so just when you're looking for these people right they might not have the DBA title um on them um but usually what I see in you know kind of larger companies like as as they kind of have many hundreds of databases to manage um you would usually under the infrastructure team you would have dedicated data or data platform team and those would be the people that know best you know what are the best practices for managing postgress if something goes really bad you know who do you call you call data platform team right right so yeah and and then you have like the with machine learning and and data science you have like that unfortunate data scientist who just happens to know a little bit more Linux command line than anyone else and then is in the unfortunate role of being the de facto like data processing nerd on the team and like has to kind of over over subscribe on on that knowledge as well and kind of coming in from a from hey I need to just like get this data processed and ready for doing data science work and no one else is doing that for me so now I got to like learn all these other things um just to get my data in the proper shape uh so we see a lot of that as well yeah I can imagine uh you know there's a sort of a theme on the internet that developers are expected to know too much you know there's this sort of general complaint that why do I have to know kubernetes now too like I'm already creating the applications I'm learning how to to package them test them deploy them and now I got to manage infrastructure there's too much stuff for me but I can remember almost 20 years ago up until even like 15 years ago where I was actually working with teams of groups of dbas like we're just talking about that their goal was to actually consolidate everything into a single database system like u in the specific case I was remembering they were focused on Microsoft SQL we were moving off Oracle uh open source had really not taken hold in the late 20 2000s essentially in in the this was in government so of course moving slower and there were trying to consolidate everything to a single tool we had multiple versions We had three or four different versions in production but we were always like let's focus everything on this particular one and now I can just imagine and this is an imagination I'm sort of looking for some confirmation Lucas but that if you're a data engineer you're dealing with a half a dozen or more different types of data like you've got you've got a lot more under your belt than just postgres and you've got to manage a lot of different things like we talked about Vector databases uh I can see how some of those data Engineers might be dealing with some sort of um oper like maybe not S3 specifically but stuff that is maybe not so much more the traditional squel sense um what do you see people coming in uh like especially when with PG analyze and stuff like that where they're running other systems as well other different tools it's like is this sort of a a problem for all of us now where we have way too many things to manage and too many tools to deal with I mean it's complicated right so I think the good like so for example you mentioned Vector databases right so I mean I think what I do see is a lot of people that use postgress end up using postgress for a lot of these different workflows right so you might be you know again 10 years ago you might be using mongod to be but then post added Jason B and so now you can use like mongodb type workloads in post right and so I think usually what I would find let's say there's a team of five you know data platform Engineers there's probably you know half of them are somewhat like well vered in postgress and they will actually spend majority of their time working with postgress right um I think like often times you know maybe what you see is like Legacy systems right like people move off from Oracle or server towards post like that's a very freaking thing I see um sometimes you know they make that over many years right so you will often have people kind of have a dual role of like an oracle DBA and then you know postgress data platform engineer whatever you want to call it right um and so they um I think like on the relational database side I think usually people try to gravitate towards one solution um I do think that you know we talked a little bit about you know like working with data and like exporting it right having business intelligence type things and so certainly you know what I do see is people using things like snowflake or click house to do like other analysis I mean snowflake is more on the you know business ETL side click house would be more on the I have a columnar store that I want to you know run um whilst for example postris has a way to store column the data it's probably not the best way to store column the data and so I would say probably the the most common pattern I would see there is probably having a good solution for Colum nner which with something like click house um and then the other thing that I do see is also um when people are like you know there a scale of let's say door dash for example I have a public case study so I can talk about this um where do dash for some of their servers uses coor DB as well right so Co DB is essentially it's post compatible but it's it's really not post it's kind of a built from the ground up way of having a distributed database and so for some of their workloads they use that and again they have a public case study if you want to know more of that but um obviously their team has to then split the attention right like some of the databases being some being cockroach yeah that's I don't think we've had a a show on cockroach DB specifically but um we add it to the list Brett yeah um but they've definitely been I mean because they were one of the earlier uh you know sort of container friendly or container native kind of ideas around uh databases real quick we're going to get let me get throw a question in here um MZ is asking what is the state of running postes ha across multicluster as stateful said is there any progress in the direction as I knew post G operator still has no support for multicluster setup I think it's a good question I don't have a good answer for you unfortunately um is the truth of it um I think in my understanding is it's not a done deal is how I look at the problem um from what I understand um this is probably something I would do research on I would look again like The Operators I would look at is the cloud native PG one the country one um possibly the solando one although that one is I would say that one probably isn't as advanced as the other ones but it has a lot of you know things under its Bel but it probably is the one that definitely doesn't support multicluster things well um I do know that cloud n fpg team has that on their road map um to a certain extent but I don't know exactly what capabilities it has today for actually managing things across clusters or setting things up across clusters um so my understanding is it's definitely still manual work yeah it feels like the cross cluster and especially anything that's cross um region the cross cluster stuff in in kubernetes I mean that's sort of been for a lot of us whether it's the Ingress and the the proxies or what every take any add-on on top of a cluster any extra controllers and getting it across multiple clusters is sort of the last piece of the puzzle I think for a lot of these I think we a couple years ago we had Contour on talking about a separate project they were working on for contour for Envoy proxy just to get multicluster setups uh and it's it's all very complicated and very hard so I I can only imagine what it's like for for databases cuz I I haven't dealt with it in databases myself either um but that that feels like that's that's like the the Utopia of I can spin up as many clusters as I want and they can all I can I can use the same tools for managing all of them and all these things talk across cluster and across region and everything's wonderful and and sadly I feel like for 10 years we've been giving these answers of like you know cross IP subnets used to be a hard thing decades ago like we're we're slowly arriving at a state where most applications including clusters themselves can actually perform across boundaries of networking and and regions and stuff like that so I'm not surprised that cross-cluster postgres operators is still in its maybe not final State um that seems that would be you're considering that oh go ahead finish F uh if you're considering that that's definitely like the hard mode right now right that's the edge of kubernetes um I think kubernetes was designed as the cluster being the main boundary yeah and uh this is we need something else that like the next abstraction layer is still being built and envisioned and kind of explored I think yeah and what I would add also I think is what what what are you really looking for with a multic claster setup right like is it so is the motivation for example to bring data closer to your you know end users right like maybe you're essentially doing like an edge kind of distribution type of thing um are you trying to you know have better ha right like so I think it's really a question of what are you trying to achieve because um just replicating things everywhere and then you know imagining that things will automatically work um kind of neglects the fact that latency is really important right so like once I start going you know beyond a couple milliseconds of latency things really will start breaking down right and so a lot of times I see people on the requirements list they have like you know cross region ha and like they want everything to just be super available but then when it comes down to you know what how do you actually want this to work you realize that people haven't really thought about the speed of flight type problems essentially yeah you still can't break physics right no matter how far as I know well and this is the balance right because like there is there is never like you're trading it's tradeoffs it's all trade-offs and I think this is part of what's happening in chat is uh Z's talking about dvas everywhere when consolidating I mean we want Simplicity we we want less to manage because that's the only way that any of us ever actually escaped I mean I was the CIS admin right and I and I escaped it into managing many more things because we simplified management we simplified architecture things weren't I didn't you know there was a day where I was assigning IP addresses manually to machines and then putting it in a a spreadsheet oh yeah because that's how we tracked IPS right we didn't have dhp yet in the in the data data center was like the last place to get the to get DHCP it was actually a security concern for US 20 years ago because we thought if a rogue machine shows up on the network it's going to automatically get an IP and be able to Route everywhere we don't want that we want it to be harder so let's I mean there's look people um we have we have challenges and if you try to make everything infinitely scalable and infinitely expandable with all of the fault tolerances at every single level you're going to need Netflix like you're going to need a Google team you're going to need an AWS team to help you do that like that's i' I I walk into teams all the time I mean I think that's one of the nice things about being consultant is you you get a perspective of all different types of organizations and everyone's struggling like everyone's trying to increase ha increase battle testing before things go to production everyone doesn't feel like any of this is good enough for them and they're always trying to make it better so yeah that's probably why you have your company it's because everything needs to be better Y and also trying to Bish right performance cost up time whatever latency um and and they kind of not stumbling in the dark but kind of exploring all the edges there and making making mistakes experimenting trying different things out for sure and I would say I mean I would also before we jump to that question I would also caution people to you know not not cosplay large organizations on necessarily right like it's like sometimes people adopt kubernetes because that's you know the thing that you should be using when really you don't need kubernetes you just need a container ECS right and so I I think you know there's this whole story around you want to be big you want to do it proper but maybe the time isn't right for that yet right maybe you should be thinking about where you're at with your own application um how you keep it simpler right versus making it more complicated uh that reminds me I was talking to a customer it was this small startup and I asked them why they were adopting kubernetes and they said because they were bored and I was just like [Music] a board startup is not a good sign and I was like okay but yeah there's there's a lot of that going on as well um well and then there's a little bit of a culture challenge here just a a little bit of a segue is like normally and I go to these cubc cons and we've been going to Docker con and cubon for a decade now or close to a decade and there is definitely a culture in the open source community and and I think cloud native is maybe even sort of a pinnacle of that because it's all open source and of course there's all these vendor booths that have quasi open source or or open core or whatever they have and the there's not necessarily a culture there of let me find a cloud service to solve this problem first it's it's it's a little bit of a culture of let me find an open source tool to implement myself to solve this problem first and I feel like I I'm the always been that person in the Middle where I stand like right on the line 50/50 and on on one side I have all these closed source sasses and on this other side I have all this open source and it's it just depends and a lot of times I don't have the people and or the time to implement a piece of Open Source when there's this shiny thing like RDS or AWS app Runner which can run containers for me so easy click click click done I don't need a cube control I don't need a kubernetes manifest I don't need all these other things um so I think and we're a little bit guilty on this show we invite all these wonderful people on to talk about open source and sasses and the that we're all that mixes so uh for those of you out there if you haven't figured out already I don't know about nmal I mean he works at AWS but for me I sit in the middle I like both there are right there are right cases for both so if you're looking at how do I save time it may not be open source that saves you time it might be a Sask product I'm not trying to make an ad for PG analy or anything like that but um it feels like one because we got someone already in chat that's a fan that says uh I really interested with PG analyze would what would bring to the table and we haven't really talked about it so we've only got a few minutes left I know we have other topics we can always get into but should we go back to PG analyze real quick sure yeah that's to me talk a little bit about what it does for those that are just joining sure um yes I would say definitely there are open source solution to what we're doing as well um and we do publish some of what we do as open source but it's more like libraries that we use um internally um so the product itself is a SAS product that you pay for which is why how we sustain a team of Engineers working on a product um you know um it's how this works um and so what PG analyst do does is really on the performance side right so it doesn't really matter um whether you're running in a container you're running kubernetes or you're running on a virtual machine um or an RDS um the idea essentially is that if you're optimizing posters performance you're trying to understand why is my query slow um what which part of the query is slow right am I missing an index um all these kind of questions that you might ask ask yourself there are ways to do this without PG analyze but you're just going to spend a lot of hours reading on materials of how to do it a lot of hours you know making making spread Sheets if you're like looking at a query workload you're trying to say across these 100 queries which index should I be using um and so really the idea is that we're collecting all that lemetry data and then um I think somebody else in a different context in the chat mentioned AI right and so we don't say AI but we do have advisors in the product as well um and the idea behind the advisers is that they essentially look at all that workflow data coming in that they make suggestions like hey change this config figuration setting or at this Index right so the idea is that we essentially make it easier for people to operate posters as a production um system be that in kubernetes or an inversal machine or an RDS very good awesome what's a what's a question that if if someone finds themselves asking this kind of question PG analyze would be the right thing to step into or to start using you mean what is the they would be asking to then come to us to solve the question yeah so like maybe I'm an application developer and I ask I find myself in a situation where you know I asked this question to myself right and then that PG analyze would be the solution to that or or help point me in the right direction I I think it's really so it's definitely not you know why should have running Post in a container I think this is a question we will not help you answer we hopefully give you some poin today the show's for exactly um and we can probably talk more about it um for you a full hour but um just you know the question that would make you come to P analyze I think is more why is my query slow right so okay like what we talked about at the very beginning I think you know why is this slow you know I like the database is this magic box right like what is happening behind the scenes and so really a lot of what we're doing right is like we build custom visualizations for example we try to you know kind of really invest into making it easier to understand what's going on behind the scenes um because I think that's really what so you know talking about AI for a moment what I think personally is even if we have advisers for example I always want to make it clear how we come upon the recommendations right so what I don't want to have is like a jet GPT that says create this index change you know rewrite this query like this and you don't even know what it was doing behind the scenes right and so I think this is kind of where we're coming from is like it's it's very much the like get the data visualize the data make it easier to understand find the right things and then in some cases also give you the recommendations awesome I think there's a good fit between uh PG analyze or those kind of tools and something like open Telemetry tracing right so if you if you've done some tracing um you've done some more advanced proactive observability on your application your microservices as part of that Trace you're likely you know your API your backend is going to hit a database at some point maybe and um this then pairs nicely with oh I find that my Trace shows that this query took x amount of minutes and it should take x amount of seconds switch over to PG analyze to figure out what's going on there right and so so the good news is so we have an open climet integration um for exactly this reasons so we can actually um as of you know September last year we can actually push spans into your traces where it will show hey here's a slow like slow period plan essentially and then it takes you to PG analy for details um and this topic if you're interested in it if any of you listening is interested in that um I actually um was in a podcast on observability cast with um um um charity majors and Jessica car from honeycomb last year um I'll see if I can take out the link so we can at least include in the absp um but um that's essentially an episode we just talked about how can we combine tracing data you know across the database and the application because that really you know kind of demystifies the database again right a lot of this is demystifying what the database is doing awesome another thing for the list Brett open Telemetry yeah we know charity uh and they a a group there we had we've had honey come on the show before but uh yeah that's awesome um well this has been okay so I've learned a lot um and also I think we could all sum it up as it's comp it's complicated but PG analyze can help and containers may not always be the right answer and do your research right I think that's what I would say it's like look at the different operators if you're looking to deploy posters in kubernetes um there are different options out there and it's really a question of researching the different options yeah awesome I didn't I didn't realize there was so many options out there so that's that's also a you know it's it's I sometimes it's harder when there's so many choices and you have to really do your research rather than just being one clear most popular you know it's got 10x or 100x the GitHub stars of everything else and you can clear that that sometimes I just appreciate that because it's like there's just so much complexity in researching and trying to figure out what the right choices are and an unbiased answers which are hard to find on the internet right um well thank you Lucas so much for being here um I feel like we've answered lots of questions and we can spend another couple of hours here Neils gets the last question by the way if we can keep this one short what is the hardest operational problem you have encountered running postgres in containers and if it's not the O killer how do you contain it I feel like that's a leading question it is a leading question and if it's the Neils I think it might be then I know that Neils has you know firstand knowledge of the problems here um but definitely the out of memory killer can have can be a big pain um the other thing I I'll just mention this since we didn't get time to talk about it earlier and I think it's just one thing it's important to understand if you're somebody who really likes extensions in containers um this is not necessarily a production problem where you have an operational issue but something you need to think about is your application developers might want to run extensions and so it's actually quite painful to manage extensions in a container in with post um so if you want to use PG Vector for example you need to use the PG vector image if you want to use post JS you need to use the post image and so it's just one thing I'll I'll mention it's not an operational problem it's more of a practical problem if you're running post a container is watch out for extensions yeah so you have to end up building your own which yeah that was number three we got the third one we got the third one yeah you have to wait till the we're gonna change the title of wait till the end for the third one yes you'll never believe what happens at the end of the show oh very good well thanks again Lucas um I'm looking forward to hearing some of those people in chat coming back and telling us how they improve their postres from learning stuff on this show um if I had postgres and production I would be improving it from stuff we learned on this show but I feel like I learned a lot um for those of you out there this will turn into an audio podcast we mentioned at the beginning of the show but also this turns into an audio podcast links below if you want to know who's going to be on the show next week it sadly won't be Lucas but we will we have a whole run of planned shows coming at you on Thursdays you can get on the newsletter again in the description to find out who's going to be on the show each week and tomorrow someone in chat ons had to remind me tomorrow we're actually having our a bonly swarm fans meet up yes swarm is still a thing and there's still fans so you can you can join us in Discord there's actually a little events feature if you're in Discord you can click on events at the top tomorrow is our swarm fans meet up totally free just come out hang out with us in video you can actually go on well sort of promote you you can ask questions you can start having part of the conversation it's uh it's like a more fun version of Zoom and then next week we have the high fivers monthly chat so if you didn't know about high fivers you can subscribe here on YouTube it's essentially five American dollars a month and you get to join us and it's like a devops hangout we we just talk about our work talk about projects we're working on some people bring their own Docker files or their terraform or their kubernetes yaml to have us all look at we Give opinions we all have lots of opinions so we do that that's next week and then we're gonna have another show next week so we got so much stuff going on do you know who's on the show next week Brett uh let's see who's on the show next week Phil Estus talking about Phil is on the show yes so so you better come back next week yes we're talk about we're gonna talk about fch which is an AWS project for running containers locally open source and Phil is coming back use it to run yeah you could use it to run postr in a contain you could use it to run post to Loop it all back yeah that's that's right bringing it back bringing it back uh I'm excited to have Phil back on the show uh like we talked about recently he's actually the only person who's been on this show that was physically here in the studio AKA second bedroom here at at the headquarters and so we figured that out but that was like four or five years ago so we're excited to have him back on the show virtually not physically here but um yeah we're going to talk about some new stuff from AWS so come back and see us awesome thanks Lucas thanks normal than you thank you bye y'all [Music] [Music] oh [Music]

Channel: Bret Fisher Docker and DevOps

Views: 1,298

Keywords: docker, kubernetes, bret fisher, cloud native, automation, containers, docker mastery, kubernetes mastery, devops, postgres, postgresql

Id: _tQ1fdI6MiM

Length: 65min 12sec (3912 seconds)

Published: Fri Feb 16 2024