What’s New in Docker Compose, 2024 Edition (Ep 251)

[Music] [Music] [Music] [Music] [Music] [Music] [Music] oh [Music] o [Music] o [Music] we are live on the internet woo how's it going Brett good to see you nmal nice nice to have you I'm I'm excited for another show with the two of us yes riing on knowledge that we sometimes have o I'm excited to learn something today from you just so you know internet we've got some AWS people too so I coming so nmal bringing the heat with the guests and I we're I'm excited because he's got some he's got some access and we're gonna we're gonna learn some things that I've needed to know for a while now and we're gonna talk about some things that I'm excited about not this show but coming soon coming soon so what is this show Brett where are we what we're on the internet and we're supposed to talk about compos today I mean if you're showing up and you didn't actually click like read the description it's about compose it's your ketchup show it's gonna not the ketchup is in like made from Tomatoes but ketchup with a C normally in this this channel every word we use has a K in it because it's kubernetes related but not today it's going to be C oh let's see if that works yeah catch there is a composed with a K but it's a different thing yes how about catching up catching up there we go we're the catchup we're the catching up show yes um hello everyone in the chat let me just um say hi to a few of my friends on the internet there Johan's here um oh look normal's here uh Ben's here from Cleveland the Black Forest that sounds that sounds like a place I I'm scared of Martin Martin and Max some of the regulars are here Alexandre all of my friends hello people and now nal's friends and Dan is here some of these people are actually from our Discord hey did you know we have a Discord it's right over there on nml's head oh am I going to get this right there we go there we go you're pretty good because after five years I still get this wrong I still go over there um and we have all these we have all these places that you can come and hang out with us so we have a newsletter you know you see the brett. live that's where we're at now that's just in case you were curious you're on the brett. live here we're there and we have a brett. newws which is not listed up there which totally should be but that's the that's the newsletter that you would have gotten last night that told you not just about what this Show's going to be about but it would have told you about two new Docker announcements that I feel like hey look we're not a news show like normal and I aren't here to list the internet's accomplishments this week but we're not we're going to do that for a second because we have we have a Discord server where you could discuss all these things over on the cloud native devops server just to prove that it's real this is it and we just had our 250th stream last week so I had to celebrate that I put a little clip out there um nmal and I going down memory lane of 250 weeks of this show so yay that was a really good that was a really good session so I I love Memory Lanes like when we talk about the old days they always you the rose-colored glasses we always think like that was so wonderful in reality back then we were we thought it was chaos yeah I like Memory Lanes if they increase my memory bandwidth wait yes so this turns into a podcast not every show only the best shows only when we have guests or there's something significant for you to learn and so that's in your favorite podcast player just look up my name look up nal's name look up you know you know I don't know maybe the title devops and Docker talk uh which rolls off the tongue like butter so you can do that you can listen in I mean one of my favorite episodes which I we need to have an I need to have an episode on here where I talk about the favorite episodes and have Clips all the great shows do that are we gonna have a clip show do we need clip show we could do a clip show let us know we had Burns yeah yeah you were on that one right no Matt Williams was here we had a we had a packed audience you missed that one so sorry you missed that one um I'm here now but you're on the D you were on the dagger show yes that was a really good one did you know you could do this you can drill down in the nmal on our podcast website and by the way he former model turned it guy um and you can see all the shows that he was a part of so if you really prefer nural over my rhetoric and soapbox stuff you can just look at his stuff in fact here in chat I will give you this link Caesar Caesar what's up Robert hey go to that link and listen to the normal episodes they're really the the kind of the best ones I mean if you kind of look at the statistics it's probably probably all in there HT you Brett H you I'm buttering you up I'm buttering you up um so so speaking of uh some of the updates um I am speaking at cubec con EU so um if you are going to be at cubec con EU not only will you be able to meet brety somewhere like around I think as we get closer we'll figure out where we're going to be but um we'll be we'll be doing a live show in the Expo so come hang out um come say hello um I I you know from our experience from cubec con back in what was that November in Chicago it seems like both of us are really appro pretty approachable so um if you're a fan of Brett if you're a fan of the show um if you don't know who we are uh come come say hello um and uh let us know what um what you're excited about at cucon EU and also check out the session I'm uh presenting with uh this uh with Shahar from um ground cover um which maybe maybe we'll have on the show I don't know I don't know yet um but uh see you in Paris um so check that out um I guess that these are all announced is that talk about the schedule the schedule for cucon the main schedule was released yesterday uh so um you can go check out uh I think I'm speaking on the Wednesday don't quote me on that don't quote me on that yeah I should have I should have had the link um uh oh there it is okay I'm trying to find it and the nice thing is is everything that's showing up is you and me on the show so uh that's a good sign yeah but there's um there's a schedule there we go troubleshooting hidden performance and costs and network traffic across multiple azs with ebpf dude that sounds like a mouthful and it sounds amazing what did you and I say at cucon in Chicago ebpf all the things like everything ebpf yeah and there's a lot of ebpf sessions at this at cucon Paris but um yeah so that's just in like a month and a half maybe like we're still working out K hotels right yeah yeah um so if you're if you're gonna if you're planning on being at cubec con um the collocated events the uh in Paris in mid-march um come find us come say hello um come hang out we'll be there yep we will and I see you Dave in chat saying please do more nomad so we got a nomad fan in there we might actually mention swarm a few times today because there have been actually swarm updates in the docker engine so I'm almost feeling like we could have made the whole episode about the announcements this week but it's two things but what we're going to do is we're going to make separate specific episodes for that that nural and I will we will learn stuff and and educate you um but besides cucon Docker announced a significant basically a whole new Cloud feature a whole new Cloud website this week they're calling it Docker build Cloud which we kind of known about for at least six months they they pre-announced it at dockercon in sort of a paperware format Early Access for special individuals before that they had had it for a couple of months internally with the docker captains this is in my opinion assuming I have still have the minutes this will be the only way I build docker containers now like especially if I'm shipping them or if I want other people to use them because it has native builders for arm and Intel so you get both architectures built concurrently without qmu which in in a lot of my cases is really slow especially when you're having qmu need to build an extension or something where it has to actually bring out build utilities it it can be quite slow and this thing has its own console now so you can go to build builder.com or in Docker desktop you have a new Builder Tab and it's slick I mean this is like the future of Docker we're looking at here where they now have a Scout website there's Hub hub. deer.com there's scout. do Scout dod.com say quick yeah say it too too fast and you get Tongue Tied we've got builder.com and in your Docker CLI or in your Docker Hub or in your GitHub action not Docker Hub sorry in your do or desktop or in your CI including GitHub actions which they already have this enabled for their GitHub actions huge fan talk about GitHub actions for Docker a lot so across all these places and more you can simply switch your context in your CLI or in the CI to build in docker's Cloud now the the sweet thing here is that you don't have to change your CI you can leave your CI in place and simply let it connect to docker's build Cloud do the builds more efficiently on bigger machines that are more native and fancier probably I'm going to guess they're probably using graviton on the back end we'll talk about that sometime in the future maybe maybe not yeah maybe um I can't speak to that so I I can't either but why shouldn't why wouldn't they I mean graviton is awesome um but they're definitely using armed native Builders and you get a certain amount of minutes so it's kind of like any other CI tool out there like GitHub or uh gitlab where you you get limits based on your on your plan and you sort of you basically start free everybody with the starter plan gets free certain number of minutes and then you can upgrade from there there's actually you know you buy into more minutes and more basically more storage and this isn't just a cloud Builder this is why it's going to change the game for me is even just as an individual I get a shared cash across everything I'm doing so if I'm building locally I'm also using that image layer cache that all my GitHub action builders that are building all my my images for my courses and stuff it's all tied into this storage thing so when you see this 50 gig shared storage cache that's for me that's just for me the individual for free and I now get all that if I switch my Docker CLI and my CI to use this cloud and it's like a on line it's like a oneline change it's not much of a change and we shouldn't we shouldn't sleep on multiarch um I think that's that was a that's continued to be a trend over the last few years and I think this year with folks trying to figure out how to optimize their cost um get more performance uh more folks developing on like you know M series Max um iot like uh yeah um we're we're hopefully going to get some guests to talk more about arm and uh and multiart so let let us know in the chat and on the count in the comments if um if there's specific topics within multiarch that that yall are interested in hearing about but um yeah that's the and I love that there's like this smoothness of using that that cloud build um like you don't have to change too much frictionless I think is the the word frictionless uh build builds um especially CU like we kind of take for granted like building and the like even small improvements in the time of those very repetitive tasks go a long way um in terms of like performance and cost and all that ja so yeah I don't build I don't make a Docker file now I mean haven't for years that I can't multi-architecture build and it sometimes like if you're a go or a rust or a c person it tends to just work I mean technically go can build for different architectures with without being on that architecture so that's really sweet I'm I don't even know if rust can do it I'm not a rust person yet um you're not Rusty I'm not Rusty but there's a lot of nuance to it so we could do a whole episode on how like the I have this GitHub repo that is a deep dive into all the different architecture types if you want to get really nerdy and exactly what the docker variables and architecture options are and how they work in terms of there's subtle little nuances that change depending on your Linux drro or which architecture you're on um and I mean it's just a lot I I did this deep dive and wrote it all up I I remember when I wrote it it was only half of the things I was wanting to write because I still have a like whip area that I never went through but this is all just like crazy and this is just one part of the multi-architecture problem and I felt like it wasn't in the docks well enough and it wasn't as well defined in terms of hey you know in some places arm 64 is actually referred to as AR Arch 64 and sometimes arm is also armhf in terms of architecture type so there's all this weird yeah but there is but at the same time like Now's the Time like it's also a lot easier I I would agree I would agree so yeah we're definitely doing a show on that this year like you and I let's write this down I'm writing on my pretend uh pad of paper that I don't have um do a multi-art show bring the knowledge bring you know bring the people it'll probably have to be more than an hour we'll probably have to like dedicate we'll have to make a special show where it's like this long format education all right sorry all righty we're gonna we're gonna we're gonna fly through the next announcement because we should have a whole show on this Docker release Docker engine v25 version 25 the next major release of Docker engine and Docker CLI and it's not like shout it from the rooftops level cool features but what are some in this week's blog if you went to the newsletter this is the week of Docker new and I don't know why I put cool pineapples but they were cool pineapples so I just did that um but I detail some of my favorite new features in the new Docker engine pick one and pick one pick one um I'm going to resist the Swarm ones because I'd love to talk about those um I can now finally specify multiple networks in a run command um so before you could spec my understanding is you could specify a network but then if you wanted to attach or connect other networks in the docker networks you needed to do that after the fact with Docker Network connect and now you can do that in the command line the one other one little thing I throw out is I mean there's there's GPU stuff in here there's re only bind mounts that are recursive now but the other little thing and I can't resist just doing one one right uh can't even behave um the start interval so health checks in Docker which aren't the same as kubernetes probes but in Docker health checks for Docker engine there's now an option so that you can basically when you're considered starting during the starting period for your container you can have the health check now go like every 3 seconds and then once it recognizes that it's available it's slows down to the normal health check interval which is pretty slick pretty sweet and that's now built in by default so I like that that's that that's a tricky area as well like health checks Readiness probes all that kind of stuff like those questions come up all the time yeah alrighty so why we yeah what's new in compose so all right compose has been sneaking out little features and releases for years now and I had to go back and the the real big change was them now granted they have been doing this actually for ever I mean I think there was maybe only like a year there in the last decade where they kind of a year or two where they kind of slowed down and went to sleep a little bit on the compos command line but when I say compose I'm really talking about the file format and the command line because they're there sometimes the feature involves both sometimes the new feature or the new option is only one or the other it doesn't really affect the the yaml spec and the CLI but for the part of this conversation let's just say it's both we're talking about both and if you go back to 2022 wasn't that long ago just a couple years they shipped V2 which was a technically a complete rewrite of Docker compose correct and you and I maybe have referred to this in the past as this is the docker space compose not the docker Dash compose totally different yeah um the the names of things over the years have got pretty blurry so um it's uh 22 was a is a was a big change with respect to that so so what was what was the impetus behind that do you know Brett like why why the major overhaul new version um they wanted so the old version was Python and originally before before compose was even called compose it was written in Python and I think if you go back in Docker engine all the way back to the early early it was also python like the original Docker people were all python fans um was it a python or bash script I feel like Docker was Python and kubernetes used to be bash and its very early pre-release like internal anyway I may be wrong internet that may be a lie um but they they they had all these libraries that docker and compose needed to have that were very similar and they were kind of creating double duty like they had to write it in goang for Docker run and Docker CLI and all those commands and then they had to write it in in Python for compos stuff and then they had this composed library for uh and lib compose and that was in in Python well they wanted to synergize which makes sense and and all the Developers for Docker are all big go so they wanted to write compose and go and honestly this was started in 2016 if you remember when we were in Berlin they announced that they were deprecating lib compose and moving to go but that didn't happen till six years later I can remember actually walking down the street in Berlin talking to Li uh Laura Taco about her the job she was working at they automated a lot of things with the lib composed library at a little company she worked at called code uh shoot what was it called it was a CI company I can't remember the name of it now it was P it was purchased by uh the Jenkins company cloudbees um anyway and they were using lib compos and that was 2016 so fast forward to 2022 it finally gets to release as a brand new and they're calling it compos V2 not to be related with the compos yamamo file format version V2 totally not related and it's now Docker space composed so all all this new effort in the last two years has been to enhance that go those go binaries and go libraries that are we can now run from the docker command line so that that company was code ship code ship that's it deep deep Cuts code ship um yeah and so there's a lot of history here and maybe people don't realize how many things composed now does that it didn't do just two years ago or three years ago so Max is saying let's go let's go oh hey thanks for the Super Chat hump gave us a super chat I appreciate you thank you so much um all right so let's just start if you want a really kind of um cheesy way of going through all this we could just go through the release notes and just scroll scroll that's not that's not what folks are here for that's not what here for this is a very effective way if you just skim it but it doesn't really explain things which is why we're here right we're here to explain things and so I'm going to start and go back in time um because I don't want where do you start do we have to do like that this is a this is a xgen reference to movie um so the we have some stuff that's experimental and we have some coming soon so I'm I'm not going to bury the lead I want to talk about like coming soon experimental stuff first in case we run out of time because we do have we are not Unlimited on time here and then we're going to go back in time and as you're watching people when we get to the stuff that you're already like ah I've heard of this I use it all day long then maybe we're g we've gone years back and we're sort of into the area that you know about with new compose because I talk a lot about compose here so so I'm waiting I'm waiting on baited breath what's this new the experimental stuff yeah so if we if we go over to I can find my visual studio and so if we go over to visual studio um we have this new thing so remember Docker space compos that's like the first thing we should all make sure we're doing because if you use the dash technically what happens now if on the latest Docker desktop is if you use Dash it's just an alias to the space it doesn't really run the old CLI that's been deprecated and removed um that all happened last year where they kind of fully deprecated fully end of life kind of thing all right so Dr compose Alpha is where you can read the tea leaves to see what is coming and right now there are three commands and a new Option o and these are things that they the reason they put them under the alpha command line argument or option I guess it's argument um is because they may be changing the name of the command or the options for those commands and they don't they don't want to have to worry about backwards compatibility so the rule is if it's under the alpha command line it could it could change at any moment and no one has to apologize um so we have this publish typical disclaimer yeah it's a disclaimer so this is like the bleeding edge experimental use at your own Peril may not be feature complete like the first one which is this is a thing I've been asking for for like six years um and it's not fully baked but we can I was actually in the captain's chat right before this asking Docker employees and other captains like hey how does this exactly work in at the end of this feature set because I don't it doesn't work like I expect it to work and it turns out it's because it's not finished it's not fully realized but publish allows us to take the compos file and ship it as an artifact to Docker Hub or your own registry and we can do that would you if you think about it a lot of other tools like Hub sorry not Hub Helm if we all think of Helm Helm has this idea of charts being stored in places and we've now had two different formats in Helm for that where we can we can store it in a super straightforward yeah easy to understand you can list charts and you can have chart um is is it chart Registries I'm trying to think there's I feel like there's another term for chart uh chart repositories but we didn't really have that with compose so now there's a couple of things in compose that are starting to we're starting to see the light of day with where we're going to be eventually able to store composed yaml in Registries as artifacts which means other people can then run our compose apps without needing to clone a repo or download a yaml file which was always the problem before if you think about giving Docker to the people to the people like that aren't watching this show that aren't fully invested in Docker yeah or just between me and you like if I wanted to send you yeah and you don't want to have to clone the repo or maybe the repo's private just because you have the source code not necessarily public or it's behind a firewall and you just want someone to be able to launch it or maybe your Docker Hub is private because you can put it in a private Hub repository but you can now have the compos file separate it and you know you don't have to necessarily put it in a separate git repo that's up to you but you can now do a Docker compose Alpha push or publish rather sorry publish the push is different publish and it packages it up in an oci artifact which I've talked about recently in our newsletter hey get on the newsletter people um we talked about the newsletter and that publish wraps it up in a consistent format and and then presumably eventually not yet but eventually you can maybe do a dock compose up with a URL and that's not baked in yet it's like the packet yeah but you can it makes sense that something like that would would be in the future right and I think a lot of us have been in a situation where we have to hunt down the compose file for like a stack of different components that are packaged together yeah and it's like in a read me somewhere it's like you don't know what if that's like the latest version yeah um don't know if it's representative of like the best way to like the best configuration I it's it's tough right now it's just kind of wild west right so this will hopefully get to a point where people start sto like writing and storing their composed files as these ENT these assets that are more canonical right and more repr and um can be shared as as representations of how this piece of uh these components need to be deployed yeah the um I I have a I think I thought I had an example but I cleared it um let's do this one but if you do you can like if I'm in here and I and in this directory I have a composed yaml file I can do a Docker compos alpha alpha publish and you can see this ex this command and when I do that uh it it it says a lot of things and I'm not exactly understanding all of it yet there's no documentation because it's Alpha indicators there's no docs where is it going where did it get published to so well Docker Hub because I'm using the dockerhub friendly format instead of the full URL I could and it and it look and what it does is it like it gives me this weird warning and then I I went and actually try to inspect this file and this in the repo uh you can actually over here I have oh there's some weird blind edit Issues new line issues there too yeah I gotta where is this I got so many windows open so this is what it does it pushes this we can't really call it a container image it's just an artifact and in fact it it indicates I thought somewhere it said something about it was an art oh yeah right here it actually has this little tag that says artifact that's the only indicator that it's not a container image it's so this is likely G to change in the future and it has a tag of latest because I didn't give it a tag I just G you know gave it the default one and I can't really look at it like there's nothing to look at there's no UI um dive doesn't work with it I can't inspect it I can't even pull it if I try to pull P it with a Docker uh if I try to go back to Docker if I just do a Docker pull it actually says uh it is not an oci image so I can't pull it into the image cache which is technically what a Docker pull command's doing is it's it's pulling down the the tar balls storing them in the image cache it has to it has to be a container image so we're we're going to they're going to need more commands and I've confirmed that they're still trying to figure all that out but what it can do today well one thing is you can do is you can do a dry run which kind of came out around the same time and dry run is starting to work in more commands so you can see what an up might do or a publish might do without actually we've had we've all had this kubernetes for years so it's finally coming to compose um and the other thing you can do with publish is something we'll hopefully mention later is the new includes feature and that's in compose yaml we can basically include compos files or link link to them refer to them inside of other compos files so you can do a dependency like you can do dependency like complex dependencies between composed files essentially and it doesn't and so those yeah and those dependencies don't have to be a file in your current directory they could be in a git repo and they can also be in Docker Hub as one of these artifacts um if we get time to talk about includes and go down the Deep dive Rabbit Hole includes could probably be its own episode on this show but um includes allows a lot more advanced complex composed stuff where if you're in a microservice architecture and you have dozens of images you could create separate you can imagine each team in your environment has or in your company rather each team has their own compose file and then you have like a master compose file with this main one that includes all the other ones and then you don't have to have a giant compos file with a thousand lines in it so interesting that's what that's what includes does but if we go back to this real quick viz we're not going to talk about viz but this uses graph viz which is basically text to images a lot of people do this for Network diagrams there's also you've heard of mermaid probably I think I feel like graph has been around a lot longer but Mermaid has sort of captured the heart and minds of the internet I feel like um I I'm I'm kind of wanting to throw in a request that they don't just include graph fiz but they also do mermaid support so if I did this viz command um by the way I I actually created this complex so you can technically do something like this I'm just going to throw this example in here let's see Brett's gnarly compos file visualized yeah so so this unfortunately doesn't work in this uh because vs code doesn't support image cat so so I'm piping the output so if I just maybe I should back up Docker compose Alpha viz gives me oh to be in the right place yeah I'm in the wrong I'm in the wrong one okay this one dock compose Alpha viz let me put it at the top there um so yeah it gives me this text to image so it doesn't really give me the image I have to pipe that output into um into the actual if you're someone who knows graph is there's a CLI and I'm saying make it a PNG file so I'm piping this in through Linux uh I give it a high DPI so it looks really pretty and smooth and then on my machine I also have another utility called image cat which allows me to show images in a shell um but it doesn't work in this shell in vs code so it will show nothing um but you know if I I could do something like pipe that out to that and um I'd have to BR that up in a browser I'm not ready for that demo um because I if I just do open it's not going to show up in this in this uh in this browser but anyway oh well maybe um maybe that's not true if I go up here yeah there we go unable to load initial data okay so it's not making a PNG that's weird it's not it's not a legit PNG oh oh you you ran it in the did you run it in the right it isn't doing it live this is why you it's a really basic image it's not exciting to look at uh the VIS is not that interesting to me so we're moving on uh but it makes it makes sense with respect to the include right so let's you want to go to include because if you then start if we're on the path where we have complex dependencies between composed files then it makes sense to start having a visualization of those complex composed files and their dependencies correct yeah the um the bigger your F like all these other someone's uh command's asking is it is the the includes similar to Docker override so there's all these options now there's at least three major ways you can include other Docker file Docker composed files with a Docker composed file or merge them extend them include them there's all these different ways and so you can imagine as people's environments get more and more complex and everything's in a bunch of different composed files we've had for a while the docker composed config command which is very helpful in that situation um because config will output and it's basically rendering all the yaml and that's not new but if we did that um it would and if I I'm in the wrong directory again doc on it anyway I I would render a composed file in all the files that it includes and then you can see how you might want to pipe that into a visualization I'm not sure if the visualization does the rendering as well if the viz command but I can see how I mean I've worked with a few teams where they have dozens and dozens of microservices and they would need this to make sense of everything what I'm not seeing in this is like links or relationships so I'm not really sure how that like a bunch of objects just listed in circles on a diagram doesn't help me so I need to understand relationships and I don't know how that works so yeah right so then there was this other third command there there's vid and then there was watch maybe yep and um watch has technically gra graduated Alpha but I think they're including it still here just so that anyone who was using it before it still works so watch if you just do um tuck ER compose I think I yeah I just spit out the commands you can see watch here so watch is something I've made shorts on this I've talked about this in other videos on YouTube uh watch I call it sync Dr capos sync but it's a fantastic way to B was that yeah not s i n k um and that one goes back a few years actually so watch is one of those things where if you're on a big project and you didn't know that there was a way around your bind Mount performance problem especially if you're talking about installing dependencies or do or building anything or doing maybe you're someone who's creating a friend end website and you have to generate a bunch of CSS or you want to Minify a bunch of JavaScript that can be really slow across bind mounts especially if you're on Mac and this watch command is essentially a Sync It's a one-way sync that allows you to to specify it in a yamama file and then synchronize files from the host into the Container as they change in real time this is also similar to something called mutagen there's also something called Docker there a third party tool I think it's like Docker syn or something that's been around for years lots of ways to do this solve this problem but now it's included in compose that's really awesome and we I think we had a question about the mutagen um acquisition um recently and and what might be coming out of that so probably something to keep an eye out for in the future around some more features in that space for sure yeah all right someone asked in chat by the way I'm trying to answer questions uh as they come in yeah um Dan saying can we deploy my Dr compos to kubernetes yet no and I'm not even sure they're ever going to do that they've been working on that in three different ways and three different projects for the last seven8 years and they've never really shipped a a long-term supportable solution for composed to kubernetes someone uh noral actually was it you that mentioned composed with a K which is just a translator it just converts but not not it's the closest thing I yes technically in the composed spec there is a way for this to work but it's I don't think they're going to ship it I mean I just keep asking I've been asking for six seven years like every year I ask and it doesn't get shipped so my my short answer is not not built likely not built into the you could technically build composed separately my understanding is you can do that can build you enable some build flags and it will enable these features and there's but I I haven't done that in years I don't even know if it's still included in the spec um it's just not there um there are by the way places like I think it's eizi and a couple other startups that have been on this show that they have tools that allow you to especially for ephemeral environments and like setup and staging and whatnot that you can deploy your compose app and technically in the background they're running it all on kubernetes but you don't have to care because you just wanted in an environment running your app and you're using a composed file to do it so there's I think life cycle does that like there's just a lot of these tools um another one we've had on the show is um octeto octeto does that um I could I could go on on there's at least three or four companies that have been on this show that can all do that um with compos files but they don't they don't go to production they're not meant to like deploy compos into a kubernetes production environment they're more for testing in Dev stuff yeah that's that's what I see it being asked for a lot of time is like the local development environment um all right so what's next we've done okay so yeah let's let's talk about something else that is not experimental but is coming in the next version because compose F compose binary versions I mean if you want to read those te Le we go back to the to too many tabs um here so technically version 2.24 of compose is not out yet but it should be out in the next few weeks in Docker desktop when I say it's not out yet it's technically available in binary format but most of us get composed through Docker desktop and so we got to wait for the next version of Docker desktop which is already in a preview release so we're expecting that in the days to weeks to come not very long and that will include V version 2.24 it finally includes Docker compose attach so question for yall out there what is the difference between run create exec and attach because those things all sound like they're letting you do something in a container but they're all different okay so let's see um run is just running the container R yeah running is like combin so what does what does create do I'm I'm this is not a test you're allowed to say you don't know you probably know so create just creates the container but doesn't start it run Ducker start with start it yeah run does create plus pull plus start plus yeah and then what was the other two that you oh uh exec and attach okay so what's the difference between those two exec runs a command in an existing container as an additional process attach connection your shell to the pseudo terminal of an existing process so it doesn't create new processes for attach attach is usually when the the reason I rarely use attach but attach is often used when people want to run let's say in auntu uh in the background they just want to run a um Debian or CTO West they want to run something and they want it to stay running in a shell nothing's actually executing just the shell is sitting there waiting for them they can do that with a run command run command-d and- it and all various options will sort of let you run something in the background that's really just a shell and the way you get back into that shell is with aach if you did a do Z it would create another shell a second shell which you may not want it it yeah so now we have that in compos big big you know that's not that's not really a but it is something I don't think I've ever hit that use case yet with respect to doer compos but yeah yeah um that makes sense Ben Ben got it yeah yeah EXA executes a command in the container so it attaches it's not oh oh the message disappeared but Ben um yeah it's not an exact of like a it doesn't create a new process so attaches you're attaching to an existing process essentially yeah yeah you have to have attach attaches to the default process in the container the one that was started when the container started exec is a is an additional process you could technically have 10 execs all running in separate shells into the same container and then you would have the PID one or the the the primary thing that was run and then all these exec processes that are running various things and if you did a PS Au you would see them all as separate processes in that container's process list and a lot of people maybe don't want that um the big reason I would want an attack is that I can see the history of so if I'm in a shell and that shell is running in the background all day long and I detach and reattached to it um I can see my history of that shell I can be I can scroll up I can do all that stuff um but I can't with an exec because an exec is a new process so it wouldn't have yeah that makes sense yeah so apparently Nigel is the only one who's ever used it that's uh and Nigel's a good friend both of ours uh he's been on this show so we we're yeah Nigel I mean that's great I mean if someone if someone finds a real good reason to use some of these obscure commands um good on them because I'm just I'm just not the one to use it um so that stuff is coming in the version 2.24 but over the last couple of years there have been these ma what I would call major new functionalities and one of those we've mentioned includes and then another one is this watch sync um MH I have actually had shows last summer so if youall are you're watching this on YouTube after the show is over go back and just search compose in this in the YouTube in this live um just to prove if I go over here and search compose oh I got to spell it I got to spell it right yeah so there's this show but I wish it would sort it by by time but yeah see right here eight months ago I did a short on the sync I'm calling it sync but it's really the watch command and it's pretty easy to understand so you can fit it in a 60-second short there I there's lots of little nuances that I could probably spend an hour on but I'm going to post this into chat so that you all later can go watch that 60c short and then even back in 2021 V2 was it was technically launched but it wasn't in Docker desktop by default yet so we did a whole show about some of the new things in V2 um I feel like I saw one are we going down memory lane again we gotta come back here's another short here's another short uh yeah I mean I've just been talking about compos forever so yeah the I feel like for a minute let's just show off the sink um yes so I need to bring and this is handy because a lot of folks use doer compose for local development environments where the there's files changing all the time but not all the files and uh back to what you're talking about like this uh the idea of like the the bind bount uh performance is impacted by not having additional information about exactly what's changing and um this kind of helps improve the performance of that use case which is like using compose as your you know using dock compose to set up a development environment locally um with you know while you're hacking away on on some kind of application or front end back end Etc right yeah yeah the the um I found this good demo here real quick this will maybe not not necessar demo but an example that kind of merges some of these ideas so um actually this one this one is just on this one is just on the include it doesn't actually it doesn't actually have the watch that we're talking about so wait one please hold so just to just to make sure that these are separate topics right include is about creating dependencies between Docker compos files um so you might have a main Docker compos file with Services backend Services um that don't change as as frequently or set up by another team and then you have your own Docker compos file for the application and the microservices that you're working on and you include that other one so that as you're standing it up you have that other team's work as well um where whereas Docker composed sync is about uh keeping a file in files inside your container up to date with changes that are happening outside the container correct yeah the the to me the watch command allows me to never need a bind mount for my source code or my source code dependencies I can completely avoid bind mounts if I want to for performance reasons it's really it's it's really a feature for perform for solving performance problems particularly on Mac or on Windows if you are uh this guy technically can be a problem on Linux too um but in Windows if you are keeping your source code files on Windows File system which you technically shouldn't really be doing but if you have a hard requirement to do that and you can't put them in the wsl2 file system which is my preference which is what Docker recommends which is what the internet basically recommends you keep those files in the Linux file system so you you're get in Linux and then you have no performance problem and there also other little benefits like you can run shell scripts properly because Linux has the execute bit but the Windows File system does not so there's these weird idiosyncrasies between NTFS yeah CS yeah and the various partition formats in Linux so with the watch command you can be on Windows MH clone into the Windows File system system run them in wsl2 Linux and every time a file changes and you can specify for these certain files I just hit my mic for these certain files sync them for these certain files like the package Json or the gy file I need you to rebuild the image and then recreate the container automatically which so you're you're giving it you're giving yeah so you're giving Docker more information about what's what matters with respect to a full container build rebuild and what matters for just like uh repple kind of loop like development Loop yeah copy in this HTML that I just edited don't re don't restart the container don't rebuild it um and it they started with these two features action the action sync and action rebuild so for the folks that are listening to this at some point in the future um we we have a Docker compos file with a section called watch and actions associated with that watch one a sync action and another section A rebuild section so what are the what's the difference between sync and rebuild here yeah that's that the sync is copy these files into the Container while it's still running and so you would do that for traditional source code things that don't have to be built right uh or installed or anything fancy run it's just going to take a file from your post put it into the same file path in the container based on your path and targets you can even ignore things and say like don't if there's anything that changes in node modules for a node app don't bother syncing them in because I'm probably going to need to do something else like rebuild the entire image because I have a new node module and that is what this action rebuild can do and I say if the package Json changes I need you to rebuild the image because as a part of that image rebuild it will then do the npm install it it sees the new package Jason so it'll get the new packages and then it'll recreate the container all for you hopefully within seconds um you know npm installs can sometimes take 15 minutes uh it just depends on with with Cloud build maybe faster yeah because you can get Cloud Builders I looked like if on the maximum plan like the business plan you can get uh 16 CPUs 32 gig of ram in just a Docker build server so that's that should be beefy enough for everyone's requirements I mean there's maybe a really obscure 1center problem of what something that they probably shouldn't be doing in a Ducker build that requires 64 gig ram but 32 gig RAM for a single image build probably is enough um yeah so they've actually added a third one of these that um I meant to look up beforehand there's a there's another need for well what if I need you to run a command like maybe I'm using typescript or maybe I'm I need to compile something like uh you know CSS I need to I need to do something minification whatever and they've now added another action type which is not in this blog post to in order to do that you can find all this out by the way the composed documentation is very up to date the doc team I see you you're doing the you're doing the great work of the internet to keep us all up to date so really most of the stuff you can just scroll through the compos docs and find you know for example let me just see if I can just off top of my head if I do Docker compose watch and then yep it's already showing me and it even tells you all this stuff like I haven't actually even read this page so I didn't I was just assuming that it existed because the docs are always so great and uh sync action rebuild and then path and Target oh sync and restart so this is the one that was is new sync and restart so for example in nodland not all programming languages are like this but in node land um if it's if you have like an index JavaScript or you have a server Javascript file like you have this initial file there and that changes node isn't going to pick up the file changes because it's already pulled in the file and so you need to restart node and that's why you would have to do something like node mod or Ps pm2 or like these other utilities there's there's also one for react that's like a custom react web server they're all meant to watch files and restart well you don't need those utilities if you're just simply restarting um yeah it doesn't run build commands I was wrong it doesn't actually yet run special commands as a part of that process but at least it will restart node and maybe you would have something in your compos file like an entry point script that would maybe do some of that stuff if you needed to for development I think by the way these three together the it creates an incredible developer experience that's highly performant solves a lot of problems that people used to have other thirdparty utilities to solve maybe even multiple thirdparty utilities um the one thing it doesn't do is a two-way sync which we keep getting I I get asked on this show a lot of times when I talk about it they want to know well what if I change a file in the container how does it sync back out to my file system if it's not you know if I'm doing changes inside there and the answer right now is it doesn't like there is nothing in Docker inherently that watches for file changes inside a container and then does like a Docker copy back out it just doesn't do it so um that's maybe a feature request I think i' I've specifically asked the docker team about it but I'm not sure if it's an official feature request if you are someone who has that you can always go to the docker road map on GI GitHub GitHub doer road map and you can request that feature and other people can thumbs up it um so so yeah so so play with this stuff we could honestly talk for another hour or two about all the little nuances if you're if you Thumbs Up by the way we got 23 thumbs up on this video right now um people in chat if you're interested in a more detailed list of everything more compos on compose let clarify on compose over the last two or three years of all the new changes oh I got we did we got one um give me a thumbs up on the video and and based on that response I will look at our newsletter which you got to sign up for brett. newws and I'll maybe put some of those more details in the list because I feel like there's probably a good 20 or 30 little things that are like at additional options or Chang behaviors that you maybe it maybe would impact you if you knew about it so potentially like a top top 10 top 20 tips and tricks for do compos show yeah there we go I like the top I like the top top 20 sh features and enhancements from 2022 to 2024 yeah so you want to watch out for what you want to include in your Docker composer oh and with that ladies and gentlemen we're gonna have to wrap this show up yes so thank you all um hopefully this was uh than thank you Brett for covering some of the new stuff in Docker compose uh uh as always check out the newsletter with some of these topics included at brett. newss and also join us at the Discord I'm gonna do this right yay yes up here at the top right um there are what was the latest count it was like 177,000 people we're over 17,000 people in that Discord server yeah that number blows my mind I don't know why you all are there but I I hear it's a cool place to be so you know yeah come hang out um yeah and um let your friends know about the show uh this gets converted into a podcast sometime in the future and um so you can listen to our silky smooth voices on the road and then catch up with the screen shares on YouTube and let us know what what guess what topics you would like us to cover in the future always open to new ideas and um we're here to find those folks and get them on so let us know we got a whole we have whole team meetings for planning people on this show so uh we and we don't know everyone so sometimes the the the the recommendations and the call outs to certain Twitter profiles from you in chat get actually people on this show so yes I appreciate it thanks normal I'll see you next week sounds good yeah thank you Brett I'll see you next week see you [Music] [Music] everybody [Music] [Music]

Channel: Bret Fisher Docker and DevOps

Views: 2,149

Keywords: docker, kubernetes, bret fisher, cloud native, automation, containers, docker mastery, kubernetes mastery, devops, docker compose

Id: sY7yFFNBDxY

Length: 64min 48sec (3888 seconds)

Published: Fri Jan 26 2024