Ask an OpenShift Admin (Ep 44): Kubernetes API Deprecations

[Music] [Applause] [Music] so [Music] you [Music] [Music] so [Music] hmm [Music] [Music] good morning good afternoon good evening and welcome to another ask an open shift admin i am chris short host of the most of red hat live streaming i am joined by the one and only andrew sullivan as well as some other red hatters who yes sir may or may not be visible right now so um feel free uh to ask us questions about anything openshift related right or red hat related for that matter we can get you the right answers feel free to email me short at redhat.com if you have questions that you don't feel like you want to ask on air that's fine too um but andrew what are we talking about sir yeah well hello everyone and welcome to uh this week's ask an openshift admin office hour so as chris alluded to this is one of our office hours series of live streams here on openshifttv or red hat live streaming i should say eventually i'll get that right so the intention here is much like if you ever had a manager or a professor a teacher who had office hours where you can come in and ask about whatever it is that you want to ask about that's what we're here for so anything that is top of mind for you anything that you're you want to ask us about whatever it happens to be regardless of today's subject you are more than welcome to ask us in chat in social media so you can reach me at practical andrew on twitter all one word and chris is at chris short on twitter and of course you can reach out via email although we usually don't respond to email while we're on stream because yeah too much too much multitasking is hard so yeah feel free um regardless of which platform you're watching us on uh whether it's one of the youtube channels or twitch if you chat with us we it gets rebroadcast across all of them and we'll see it and we'll do our best to respond so today as chris mentioned we're here to talk about kubernetes api deprecations uh thank you for posting that link chris you got it no problem uh so this is an interesting one and we've talked about it in bits and pieces over the last uh what month or so we we've brought this up and talked about a couple of different aspects i think the first time i mentioned this was in the what's new in 4.8 uh when we talked about the api request counts api so this is kind of an extension of that and as we get closer to 4.9 you know this is going to get more more important and more relevant so we we wanted to bring on the subject matter experts the folks who know far more than i could ever hope to about this and of course that starts with rob so rob if you don't mind introducing yourself hey everyone happy to be here i'm rob sumski i'm on the product management team for openshift i've been in the cube space for quite a long time i deal with a lot of things being built on top of kube apis and so api deprecation is part of that but talked to a bunch of folks in like the operator community and other folks building on top of the openshift platform so yeah happy to be here thanks yeah rob i'm surprised this is the first time we've had you on this on the uh stream because like you you came from korres you have a long history um i say came from you were you were brought over with core os um so you have a long history here and it's it's always fun to to chat with you about that yeah been in the cube space kind of since it existed so yeah yeah so we are also joined by and i will apologize in advance because i'm terrible with names camilla camilla macedo yes perfect they spoke perfect uh yeah i am software engineer i have been working with the operator framework for a while i believe a little bit more than two years and i'm happy to be here and help you if you can and last but certainly not least is frederick yeah thank you andrews uh plays part of the openshift engineering team with camilla i've been at trader for seven years and a bit more than 20 years in 19. thank you and you you know it's a serious topic when we bring on both product management and engineering that means that that means that uh you know chris and i are way out of our depth and and we need help quite out of our but you know working our way there [Laughter] uh so i i don't want to waste too much time uh it's an important topic i want to make sure that rob and camilla and frederick have plenty of time to talk about it as well as answer any of your questions so i want to go ahead and get right to our top of mind topics for this particular week so for those who have watched the stream before and particularly for those who have watched for a long time thank you you know that we have this kind of recurring segment if you will which is things that chris and i have found that are important or relevant that have happened in the last usually the last weekish or so but they bubble up and i think that they're important for you all for our audience to be aware of uh so the first one which is a little bit self-serving uh is there will be no stream next week uh so i think and chris maybe you know um i don't know how many streams are happening next week but next week is ansible fest and i i have the privilege of being one of the uh subject matter experts so if you're registered for ansible fest if you go to the connect tab there is a brain date thing and you can go and register for brain date i'm going to call them appointments not to make it awkward um yeah have you not heard a brain date no i've not that's it yeah service name so yeah essentially you can go and schedule one-on-one or or group appointments with with other folks so the first time i heard about it was back in uh i want to say kubecon in like austin in 2017 or maybe it was was my first cubecom it was then openstack summit maybe one of the events in austin was when i first heard about it but so yeah um if you're attending ansible fest uh by all means go and check that out there's more than just me of course there's a lot of folks right so yeah not not just me you get enough of me maybe maybe talk to somebody else yeah so they're actually streaming a lot of the uh ansible fest stuff this year on the red hat channel so just stay tuned for that um but yes we'll have an augmented schedule next week on the 29th and 30th as a result of ansible fest so yeah just keep that in mind indeed and uh i think all of that is on the streaming calendar as well and uh should be yes i'm going to trust that you've posted a link and to all that stuff if folks don't already know how to get there so the next one that i want to talk about is another one that we've been talking about for quite a while which is the stable upgrade edge for openshift 4.7 to 4.8 so we've gotten a number of questions about this both uh internally and externally thank you to everybody who has sent in messages um and we are now as of i want to say monday may maybe tuesday at the latest we now have a stable upgrade edge from 4.7 to 4.8.11 i believe is which yeah 4.8.11. uh so congratulations we made it we got there um engineering fixed a virtual hardware yeah so so it's funny right um it felt like and i don't know why this one felt like it took longer to go from you know ga release hey you can release to there's a stable upgrade edge but engineering actually sent out a really interesting email where they were highlighting that with this new process which we've been using since 4.6 this upgrade edge was right in line with the others so the upgrade promotion happened 55 days after ga and included over 120 bugs since that ga right including i think five of those were upgrade blockers um so 4.6 was promoted uh to stable after 58 days and 4.7 was promoted to stable after 50 days so this is right in line like it seemed like a long time at least to me i felt like i was constantly you know hearing about and and people are you know asking regularly about when it'll be available but really it's the system is working it's working the way we want it to and the way that we intend it to so that you know when you use stable which is something like 90 percent of clusters uh uh use the stable upgrade channel um when you use stable you can be confident that you're not going to encounter any uh any edge cases or anything like that uh i'll throw in a plug for please please if you've got a cluster that you can add on the fast channel it really helps us out um like you know we did find a few blogs that we did want to block upgrade edges and you know it's not that the whole platform has an issue but like in very specific combinations um you know something needed to be blocked there and so we did it to protect you um so yeah but uh so trust your uh cluster console or the any of the tools we have to visualize that graph because when they say it's okay it's okay um but then uh yeah we need folks to mix in some of their development environments their test environments um if you've got like a a scale testing kind of environment um throw your cluster on there and get us a little bit of feedback on that it really really helps yeah and i'll also add that fast updates right if you're using the fast channel it is fully supported there's you can absolutely call you can open a support case you can help with anything that happens inside of there it's only the candidate channels that are are not supported i've got it that's good i've forgotten that over the years i feel like so uh a shish how to tickets take storage at cluster level do you mind just if you can clarify yeah um on that um so i'm going to share my screen real quick here [Music] nope that's show notes that's a cluster that's the one i want um so what i want to show here is uh so last week late last week the kubernetes team announced a couple of cves um so i saw these over in the kubernetes slack if you happen to be a member there check the announcements channel you'll see that that shows up so the first one is this 2020 8561 so this one it is a flaw where if i'm remembering correctly essentially using the validated web hook configuration and logging above a certain level it can expose certain information so if you look at this guy you can see that openshift 4 is not affected so essentially congratulations we don't have to do anything and that is a result of as you can see here we don't support logging levels higher than eight and the vulnerability only affects when you have logging levels set to 10 or higher so the second one that i wanted to touch on and this is the one that i've heard more about is this 20 21 25 7 41. um so this one and uh let me check to make sure that we're yeah you're posting the uh links yeah yeah thank you uh so this one is or it does affect openshift4 you'll notice that we don't have an errata or a release date here uh so this is being worked in the background once that becomes available we will make sure to make it you know out through the standard upgrade update mechanism for all affected versions one thing that i would highly recommend is somewhere on this page there is a follow button i thought um it might not be there because i'm not logged in but but you can i believe that there is if you're logged in you can click the follow button and it will send you emails when this page is updated including when this errata happens you can also go over to the bugzilla that's associated with it and you can follow the bz and that will announce that will let you know when there happens to be any updates or changes inside of there so definitely this one is a pretty serious one definitely keep an eye on that one and make sure that you are updating your clusters as appropriate um the cbe does note that it is i don't know it's not fully mitigated but it is maybe less affecting because we use sc linux so we've we've had this conversation before se linux fixes or it doesn't fix it stops a lot of things that uh affect other platforms so yeah and is it true that we're the only major cube distro with seo linux enforcing out of the box i think that's true i think that's true i think so i think there are some others that use app armor but i know dramatically less about app armor than i do sc linux naturally right i mean to my knowledge yes so anyways we put a lot of hard work into that so i'm glad it pays off uh you know in situations like this yeah and you know that in [Laughter] well that in uh in combination with things like you know no root we we force no root containers you know we you know sccs and all the other security mechanisms that we have um where's that security e-book i always like to post that one oh yeah i should have had that link candy i should have that like as a short command duh um so while you're digging on that um the last thing that i'll touch on here is uh so last week we had christian on christian talked about windows hosts bring your own windows nodes to openshift and he alluded to it would be released as generally available soon and sure enough uh monday it was released as generally available so if you have any need for windows nodes and you did not or you don't are not using one of those previous platforms specifically on-prem it was limited to vsphere ipi now you can bring those with any upi deployment there's a couple of blogs here so this one is from anans also christian has one uh be sure to read through these be sure to read through the docs watch the stream from last week christian walks through all the prerequisites any kind of other gotchas and other stuff that you may need to be aware of but yeah windows nodes windows containers go forth and do great things [Music] uh and i just dropped the link to the security guide is that the book you were talking about yep yep thank you okay cool just making sure and we also have a lot of other ebooks folks especially about operators which rob mentioned so feel free to grab those if you need some reading material as always so i see lsi uh is it possible to deploy two openshift clusters on the same domain so yes the base domain is okay to be the same but remember the full dns name that it uses includes the cluster name so you can have cluster1.basedomain.or you know tld cluster2.baseddomain.tld i don't think you can have like you can't have two clusters that use the same cluster name or the if you um and it sounds like you're saying here like can i change the route to use oauth.openshift.baseddomain or api dash in base domain i don't think you can do that with two clusters simply because it needs some way to figure out which one to route to so you can change some of the host names that we use as a day tuition but they still need to be unique just so the routing works yeah yeah i mean i don't so it's here's the response the subdomain is the last option my apps need to be exposed on the base domain not on the sub domain so the wildcard start will be an issue well you can still do that can't you with like a dnsc name i i wonder if you could use an apps domain oh that's true because so the app's domain and i just stopped sharing my screen let me bring that back so let's go to the docs [Music] so we want apps domain so the apps domain is uh you can tell i go to this docs fairly frequently so it says for um for aws it actually works and is supported on any deployment type but effectively what this does is when you create a route it uses whatever this apps domain is for all of those routes and that domain endpoint so this apps.acme.io that dns endpoint can be located on an external load balancer or wherever it happens to be so that could be one option where that would get weird is that external load balancer would still need to understand like app1.ac.apps.acme.io needs to go to cluster 1. app2 needs to go to cluster 2 right and so on and so forth so the best option in that instance now i'm wondering if it isn't to use a ingress controller from one of our partners so think like an f5 or a citrix adc where it's able to reach out and control right the the operator and the integration is able to reach out and control that external device that may be the most robust option that you have um so i'm catching up on chat drop that link real quick yeah i'll put the link to this um apps domain inside inside there the apps domain usually this comes up in the context of um with on-prem ipi when folks want to use or want to move the apps star.apps endpoint off of the virtual ip the vip that's managed by the cluster and onto an external load balancer this is one of the ways that you can do that awesome without yeah without changing significant things inside of the cluster it's a very good point all right i will stop my sharing again and i will uh i'm that's all i've got i think um i don't see anything else in my notes here so yeah let's talk about api deprecations so so rob you know we we chatted about this beforehand i think you've got some slides that kind of talk at a high level of you know why this is important why it's and why it matters to us and what it affects so i definitely want to make sure that we address that because i will admit um you know this is the the ask an admin you know live stream and we're talking about apis which not everybody you know some people are like what that's you know apis or developer things well no this is important for all of us and that's why we're talking about it yeah so as you mentioned i do have a few slides i promise they're mostly just to show you some commands that are hard to describe verbally and some screenshots so we're mostly just going to be talking so i hope i didn't scare you away already um so yeah let's jump into it so basically um in kube uh they have a api deprecation policy um and uh they're actually very generous about this so uh very generally like some of the apis that we're going to talk about today i'm not going to list them all up because there's a ton but there's a good link for them um have been like deprecated since like cube 1.6 like we're on we're talking about 122 here 1.6 like years ago um so you shouldn't be using these things anyways but some of them are a little bit newer and so they're finally removing a ton of these in 1.22 and so that's why we're talking about this today this also corresponds with openshift 49 which will contain kubernetes 1.22 and so you can kind of use those interchangeably as i will today um so let's talk about a few of the popular ones that are getting deprecated and what deprecated means is when you submit objects in the old format the deprecated format they will either just straight up not work and your cli or whatever tool you're using will just say i don't know what that is or in some cases we can actually uh do an upgrade to the actual like transform it to the actual api the real one like if it went from beta to stable um right so sometimes that can happen so like i want to drive that point home very hard right like deprecation can mean it's gone from a beta state to a ga state right that just means you basically just have to change one line to use the new api um that's it right sometimes that's as easy as that other times there's an actual change of foot like in the example of pod security policies there's a new thing in place for pod security policies um like docker shim right like it's not a problem that it's being deprecated because you have four or two more releases now i think before you have to actually get off of it um so that deprecation process is takes some time and once it's done they usually say four releases but sometimes that's not the case it's longer once it's deprecated though and you stop seeing warnings that's when you know you need to have your stuff up to snuff so make sure you check out those notes and specifically for us we're talking about operators today right yeah so i was going to cover there's kind of a few different audiences that need to care about this per uh andrew's comment earlier so the i want to talk about operators because it's content producers if you produce a helm chart or an operator or you've got a tutorial online or whatever that's using outdated uh resource yamls um you need to update that or it's not gonna work so there's that side of it and so we've got a very rich operator ecosystem work with a ton of partners building some really cool stuff running uh databases machine learning workloads all that stuff on top of kube and so if under the hood they're using one of these they need to update to not use it and again this should have happened you know over many many months and releases because this has been planned for a while the main one that you need to care about especially in operator land is the custom resource definition which has been a kind of v1 or a stable api for a while now and so they're finally removing the beta version of that and chris as you mentioned so the spec of this thing did not change we basically said hey the beta 1 is good let's roll it on up and make it official so all you need to do is just change like basically get rid of v1 beta1 on that um some of the other ones are a little bit more invasive because some of the the spec actually did change like either field name went from not required to required or change names and that kind of thing so some of them are a little bit more i listed only four of them here there's actually if you um there's a link i think that we have um in the show notes for the full list on the cube site upstream um but these are the main ones that i think folks are probably using out there in the wild um validating and mutating web hooks are useful for admission controllers and we actually use them in the operator ecosystem to put mostly we manage the tls on these so that you can do expose some metrics and other things and so that's where those get a little bit more widely used in our tools and then certificate signing requests for folks that have built automation around some of the cert management in cube you know openshift kind of does this for you by default and you don't really have to mess with it too much but if you do want to you know start monkeying around with the cas and and do some things um which a lot of enterprises do um that api is changing as well so rob can i can i ask a primitive question so you mentioned before like the apis are version so there's v1 alpha 1 v 1 beta 1 beta 2 beta 3 and so on and so forth what does that actually mean and generally do i need to pay attention to those um so i think the the first lesson is yes you do because of situations like this but for the most part that's just an indicator of like um if i'm going to be building something against this how how on top of it do i have to be if you're building against like an alpha api it could mean that that never progresses and like you're building on something that might just immediately be removed or like everyone will stop working on it and will pursue do a different so i think there's like the maturity aspect of that but then technically when you were writing against one of those things you were fulfilling the exact spec that that api provides um and so you probably if you want to take a new feature in kube that is in a newer version of that api you have to update it and update all your code to do that so you as an end user it's pretty easy if you've got like a get ops repo and you've just got your yamls in there to just do a regex or whatever and change stuff and da da da you're good um where it gets harder is us on providing the platform side we we do a bunch of upgrades and manage you know deprecating and enabling these new apis on your behalf so if you're just using openshift you're good to go now if you're a content producer and you maybe say you support a hundred or a thousand customers that are using an alpha api oh you gotta have a plan there for how to get that from alpha to beta you know in an automated fashion unless you want to like really just disrupt your customers so that's where you need to pay attention to that so i think most people hear alpha beta and they think you know early stage or um you know unstable is probably appropriate it means that there can be changes you know significant changes to that api endpoint but i also think a lot of folks think unsupported um so is there like if i'm using an alpha api for something in open shift does that mean that i'm doing something that's unsupported um it kind of depends um we have a few situations where changing around some of the apis is really invasive and so we have actually we do fully support an alpha api but when we get those from upstream we typically would follow a typical alpha beta stable and not support the alphas we actually usually don't enable a lot of the upstream apis until they hit beta so you usually won't find one of those in an openshift cluster unless you've specifically toggled on like a feature flag or something like that got it okay i'll stop distracting you now it's all good so that was kind of the overview i've got a few uh kind of like commands and screenshots so we talked about content producers but there's also just admins that are running clusters um and you probably just want to check to see hey my users on this cluster what have they installed what are they actually doing so if you're if you produce an operator and you maybe even use our operator sdk so the easiest thing is to start grabbing for some of the apis that you know you might be using and just go check their version so like api extensions v1 beta 1 this is the old crd api that's being deprecated you can also check for any of this and more with our operator sdk cli you can validate what we call bundle which is a bunch of the metadata about the apis you use um and say hey validate this against cube122 and it will go you know give you some output for things that you're either using or not using and what you need to do there we've got a bunch of cooler tools for um live cluster usage so this is think operators it's more about clustered content right it's like other part parties producing stuff that you're probably just using um but what about live stuff i've got somebody that's got some jenkins job that's you know using the cube api or something like that so in openshift we have two different things you can do first we have an alert that fires when you're using an api that we have marked as deprecated and then that will continue kind of firing um and increasing severity and to as you get from like openshift four six to four seven to four eight for example if four eight is when it's going to get removed um and here's some screenshots this is what that looks like and so in the message you'll see that the group version and resource that's being used is the the actual object and so we output that so on the right hand side a few days ago i grabbed this screenshot of a live cluster and it's using the mutating webhook config api that is being deprecated because it's a v1 beta 1 not the stable version of that and so you can go see exactly how many of those are firing for all the different apis so that's a really cool thing that we've got and we can go poke around on a live cluster here in a second then the next thing you can do is actually use this object called the api request count so we've got um this is kind of a custom api that openshift has to track other apis and so what you can do is go look at specifically one of these and go see which release it's being removed in and then what is the actual usage and i i kind of put the table version here but we can go look at the live one and it'll show you by um service account who's actually using this thing so you can go yell at this team or that team for you know not updating their code to use one of these new things um that's something i'm interested in because we we've shown this oci get or oc get api request comp before but i was never sure if we could see like what actually is using those apis so that that's something that's interesting to me and the fact that it lists out what's removed and which release i think it's awesome yeah trying to make this really easy for like this to kind of be self-service too so you know you as an admin made people say hey this is my i just provide like a multi-tenant cluster this is my tenant's problems um okay great well they've got a self-service api to go figure out what's happening and you can also use that same api to like you know really get after somebody if you've given them a lot of time to migrate so i i have a question there of let's say that i have a an application team that's using i don't know pet sets right i think pet sense was actually removed back in like kubernetes 1.4 or something 1.5 so let's say they're using something that we know is going away can i if i update the cluster will that break them or do we prevent a cluster upgrade if that api is in use yeah great question so um yes in openshift we do in upstream kubernetes there is not a facility for this so specifically especially in our operator ecosystem openshift will notice that hey you're using an operator that maybe is just not even present the partner has marked that it isn't available in the next version of openshift yet and so we can say hey you probably don't want to upgrade because you probably care about what that operator is doing or the api has changed between this operator versus that one and you know there might be a change required there or you need to upgrade the operator before you go to the new version um usually like our operator authors will write things so that they work across like um openshift four seven four eight and four nine and so then you can upgrade to the new version one for eight before you then jump to four nine that type of thing but we've got kind of uh warnings and like walk-throughs for all of this and i i may be getting ahead of ahead of us here but is there any mechanism that will kind of not just detect that that api is in use but automatically kind of update the object or redirect to the correct api um yes so you actually see us do this well you probably don't see us do this because it happens transparently in open shift is we will do automatic kind of mutation of older objects into newer formats just as a kind of service that you get for free um and so i forget i think i um i've been using a single deployment yaml for my personal website on a cube cluster for like a really long time and i wrote it like literally like four years ago five years ago and never updated it and finally um the uh it stopped working when i needed it i made a change and wanted to redeploy it and openshift i think it was like the either the alpha or maybe the beta of the deployment object itself and it was like i don't know what this is anymore but what's cool is my website kept running because openshift had automatically dragged that thing forward through all the api changes because it had that logic built into it so i i needed to update my yaml before i could apply it again but like it never stopped running and you know because openshift carried it forward which is kind of cool yeah so i i don't know i thought you said you might have something to share so i'll i'll let you i don't or a demo um i don't know if you want to switch that um i see that there's a question so a shish clarified looking for or asking about backup strategies so i'm going to paraphrase here of i have a cluster and i want to back up the entire cluster and then restore when needed that's a big question yeah drop some links about you know the cluster itself backing up at cd and about disaster recovery but there's a lot of moving pieces there and i knew you would touch on this andrew so let's go yeah and so i also i want to highlight that we also talked about this in one of our streams um i'm going to try and find which one that was um it was relatively recent um anyways so we talked about it in more depth in one of the stream one of the recent streams chris will find the link for that and post it in there um so i want to highlight a couple of things so an etcd backup is it includes everything that is an sct so what's an cd it is the state the expected state and the desired state of the entire cluster and that includes not just things like hey i need this deployment to be created and it needs to have three replicas and it needs you know i need this service it also includes things like i expect there to be a node named you know xyz and i expect it to have this type of configuration so where this gets weird and risky is let's say that disaster truly happens right and all i'm left with is an scd backup if i create a whole new cluster and then replace that fcd database it's going to essentially it's going to think it's the old cluster at that point and it's going to go through this chaos that may or may not be successful to try and make the new cluster look like the old cluster when what you really want is to redeploy all of your applications inside of there right so you don't necessarily want a whole sed backup you just want a backup of the objects that are relevant to your application um so oftentimes this is why like during that stream you probably if you listen to it you'll hear me say like cd is not a backups are not a disaster recovery you know strategy that's why i posted the two links yeah and when we had who was it i think it was a nons on when we we had a stream dedicated to scd when we talked to anons you know his thing is and i talked to one of the support people as well you know scd backups are great when you lose a control plane node or two or three but the rest of the cluster is still there so that you can bring back that control plane with a minimal amount of fuss so scd backup's not the greatest strategy in in all scenarios but are a possibility in some scenarios just be cognizant of that so what's the real answer here uh so there's a number of partners that have capabilities here i want to say trilio comes to mind um veeam there's a handful of others you can look on the the catalog or on the marketplace to find all of those another option is something like valero where you can go in and you can use valero to back up those objects and one that's and rob i'd be curious your your perception here as um you know the the the pm who is managing the pm cluster or kind of is the face of managing the pm cluster get ops um you know get ops is like it forces you to document and use like this state of my cluster and a very um i'm going to say heavy-handed christian's probably grimacing but it forces you to do that so that if something happens to your primary you know cluster great deploy a new one wherever it happens to be at and then just tell the get ops operator you know hey deploy this application deploy this stuff make it available in in the new cluster yeah get ops is great for that it's something that we do use on this cluster where a bunch of our product manager all share this same thing and so if you want to onboard an app yeah you go through get you make a pull request and all that um so we use openshift get ops and argo cd to make that happen but yeah it's perfect for especially oh go deploy this on five clusters or like i want to give this to somebody to run on their local uh kubernetes and docker or something like that like here's all the manifest you know if that source is an ncd and ncd is gone that's not a great position to be in yeah so uh shish is asking if the team can email us yes feel free short at redhat.com and andrew i don't know i forget yeah i just typed mine in there first name.last name for me yep cool cool uh should we poke around on some uh yeah all right so um first thing i thought we could look at is um that custom resource uh for the api request count and so i'm just going to search for that really quick so you can see it shows up here one cool thing about this is you can see every instance of basically every api that you have on here and so i've got some operators installed so we've got tons and tons and tons of these things um but let's go just like search for like let's say pods right let's go find just like the v1 pods api which is right here because we should have some decent amount of usage of this so if you go over to the yaml section here um you can start seeing like you know by hour by node by user by verb um it looks like our prometheus that we use for openshift monitoring is doing 400 some watches um which makes sense it's looking at all the state and the scrape targets and all that stuff so you know that makes sense and you can keep on going down and down and down um we can see here oh this one is actually a kubelet is doing something here um so you can see uh that it's doing um a number of different deletes gets like i mean it needs to run the pods and delete the pods and all that so that makes a ton of sense as well and just keep going on and on and on um so that's how this is useful even if you just are just trying to see who's using one of your apis for knees this is not like necessarily like an auditing mechanism like i think you could maybe use it for that but like the audit log is going to be a better thing for that um so let's go back here and we'll look at i'm trying to think so we've got web hooks is one of our api so you can see here that we've got the v1 and then we've got the v1 beta 1. so this is the one that's deprecated this is the one that's in use and so we can go see how much this one is being used here so it looks like not a lot of usage on this one actually so you can see some of these nodes um are not doing any usage so but we've just got them listed out here and so great we're not using this one it's going to be removed in one two two and our request count is zero so you know that's perfect we're in a golden spot here and so you can go check kind of each one of these things um let's go see maybe if we've got a certificate signing request maybe get a lot of these um it's not the cert manager one i don't remember i think it's this well that's the v1 so it's probably this one um ah so okay so we got a few of these so cube state metrics is doing some watches and did one list of this um and so um what's interesting about this is this will get picked up since this is a part kube state metrics is part of openshift this will get picked up when you upgrade from four eight to four nine is we'll upgrade this one for you so that one is not to worry about but if it was anything the non-openshift in here then you really start to care about that so looks like we're covered on this one as well so that's kind of how you can use this object to start hunting down some stuff and if we have so of course application teams right a developer can create an object that's using one of these what about um like non-red hat operators um maybe partner operators or or you know operators that we you know you've made yourself i assume that let's say it's it's a partner you would want to contact the partner and basically you know hey we're seeing that you're continuing to use this api we know it's going to be gone and 1.22 have you fixed that yet is that the proper course of action yeah because it might be oh we actually have a pre-release version of that out if you'd like to test it or whatever like um because one of the interesting things about you know we're kind of talking about openshift here but this is a kubernetes problem so if you if you have a partner that supports all kinds of cubes they all have to deal with this openshift happens to ship kubernetes releases on a pretty consistent basis basically one release after they come out upstream so you know roughly about a quarter later and so we usually see this first before other folks and so it might be partners hearing this for the first time possibly because of openshift users but then it's going to be other managed service users once you know pick your next favorite distro that updates then all these requests start flooding in so it does help to engage your part and maybe even help them test out they're already working on many of them have already operated like like we said this these changes have been in the works forever yeah i think it's important to highlight um what you said there at the beginning which is this isn't an open shift thing this is a kubernetes thing so it really this is affecting everyone it just so happens that we're making a a big deal out of it quote unquote uh because you know we are relatively you know early compared to many distros with the 1.22 adoption um i also had a random thought of like if before releases were quarterly when we moved to three releases does that make them like trimesters i guess i i don't know what's the terminology there uh three a year [Laughter] i mean you could say trimesters but um my wife isn't home right now but she probably just gave me a dirty look yeah um it's not as enduring as the normal use of trimesters i feel like no um the last thing i want to show is one of those alerts um if we can go see so we've got you can see i've got a bunch of instances of this firing one two three four because we're using some apis that are being removed and so for each instance of this that's firing you can actually go look at some of the information about what exactly it is um and then if that usage did tail off because oh hey i'm gonna do a deploy tomorrow morning that's gonna fix this you could see it drop off at nine o'clock in the morning or something like that as that specific api stops being used this one specifically is the v1 beta 1 of the ingress api i'm going to take a brief moment to point out because i always remember this only when i see it in your cluster there you have that big purple banner about it's being fully get-ops driven one of the upcoming streams that we have and it keeps getting bumped out for various reasons um is around customizing the console and the stuff that you see here so we'll be talking about that in the not too distant future yeah so that that's a console customization that again is run through the actual get ops pipeline that it's even telling you about so you know you can customize it with a yaml object and so one of the things that we when new pms want to get familiar with the process we say hey go change the color of that banner it used to be blue now it's purple somebody else might make it green um and that's a way for you to see how the process works and you know 30 seconds later banner's different so very cool um so our hope nine asks uh are there needs to have and now my chat just scrolls um are there a need to have special processes for the eus releases as they could possibly hit a lot more deprecations once they get out of their 18-month hiatus i mean that's a very good point yeah yeah so it's probably not like a a special or a different process it's just that you're like compressing a bunch of changes into like one like big event because you're going to go probably from one eus to another eus and eus stands for the extended update support so this is just a longer period of support than we typically support a release and that's to give folks more predictability so we've got customers that might deploy openshift on like a cruise ship that doesn't come to port for you know but every year or something like that where they actually get done yeah yeah when they want to change software um and so this is perfect for that use case but many other things too you've got a factory that's really important that you know can only come down at very predictable times like you have a shutdown over the holidays or something like that um so uh yeah basically what you're doing is um these alerts specifically actually mentioned the eus so that you can get a heads up on planning for this as part of that eus upgrade because probably what you're going to do is you're going to upgrade openshift but you're also going to upgrade all the software that's running on top that's the actual you know good stuff that meets your business goals um so in the case of like that cruise ship you might have some like the apps where you like you know order your food and like all that new versions of that need to update these apis if they're using them and if you've got vendor software that also comes on top you need to make sure that vendor has also updated their software and so these alerts just help you try to get on that kind of spectrum of thinking because you typically most folks when they do an eus upgrade they don't have a lot of wiggle room and they just really need to have it planned out very well because especially if the cruise ship needs to leave it's going to leave port you know after like it's three months or whatever if it's downtime while they upgrade the machinery and stuff um so you got to get it done in that three months one one thing that i think that is very nice about these alerts is like it's possible the admin prevents problems like when you upgrade to the cluster right so if i have a lot of operator installation no matter if it's from the partners i can go there and checky if he diverts my installation is using the deprecated apis because if he is using enjoyable the cluster for far nine it will not work right so it's a very nice point uh it's a very nice feature to to help me help the items check that right yeah and our eus cadence does just happen to roughly match the upstream api deprecation periods so if you think about if it's four releases upstream and we're going to go to three releases a year we actually have more overlap there and in some cases we actually we don't plan on extending um our own support for things that have been deprecated upstream but we have done that for certain situations and so the main point is we've got your back especially on those types of special scenarios where you do need just to make sure that that upgrade goes really well we want it to go really well for you exactly now don't be sorry canelo you're fine yeah um like i said at the beginning of the show the engineers are here there you go that's someone saying where i came without your voice it's me [Laughter] sorry folks my voice is terrible no you're fine you're fine i think it's more the the people can't see you and then all of a sudden you appeared yeah i was hearing thunder here so oh okay yeah it's going to be one of those days here in raleigh it's just going to be rainy here i mean too ugly so i i try to not appear very ashamed person so just changing it you know fair enough all right so you have to switch back to that slide really quick because i realized there was one thing i didn't talk about which is um we mentioned some of the auto conversion that the platform does but also if you use our lifecycle manager if you're an operator author you can do some auto conversion of specifically these two apis the validating and mutating web hooks because we just we use those as part of our sdk we've have a smooth upgrade path for you so there's an sdk command that you can use to do that um and basically i've kind of mentioned this before you just get this for free because this is how we do our um some of our tls management just as part of the platform is you know we have to secure your web hooks and so we want to move them forward in the apis as well so another reason to use the lifecycle manager which is built into openshift this is when you go click uh you know install mongodb this is what you're doing behind the scenes yeah we we had a stream on uh olm not too long ago as well so cool i will find that as well yeah you you have i we each have different things up which works out well um but i don't have the list of uh um so i'm just kind of reviewing the uh the set of things that i had um kind of pre-staged or or the different questions that i had thought about coming into this um i think the only question that i have um at the moment is what's the best way to be informed of future api deprecations um that is a good question i would say i mean we talked about the alerts and stuff but that's a little bit like kind of reactive and so to be proactive it's really look at what the upstream is doing and you know we'll talk about this stuff on the openshift blog as well and you know we're obviously tracking that stuff very closely but i think that's where you're going to start to see you know if you really want to be really bleeding edge like the change about pod security policies and its deprecation was talked about um inside of the you know the working group long long before it actually became an official policy um so if you want to really get ahead of stuff getting involved in efforts like that is how you do it yeah i mean i would recommend folks like subscribe to the changelog somehow right like there's an rss feed there uh the release notes on the docs as camila just mentioned is also a good place um and also that deprecation guide uh that list versions for each kubernetes release so yeah nobody nobody reads the release notes actually i know that it's very smart for me it feels like sometimes it's like ah actually that brings up an important point um so a couple of things that i wanted to highlight um so first going back to the very beginning when we were talking about update edges so it's a little known fact that we actually publish the blocked update edges as errata so let me share my browser real quick here that guy and if i go to um i'm just going to search here because i don't remember the url off the top of my head but if we go here and search for open shift i want red hat open shift container platform you can see in the synopsis here it'll tell you about all of these i'm going to scroll back and find one but in these in this errata it will tell you when there are blocked update edges right i should just search at this point but anyways 48z upgrade to 4859 there you go there we go blocked upgrade yeah so if you watch these errata it will tell you when the edges get blocked and it will tell you exactly why they're blocked and all that other stuff i know historically for anybody who's watched the stream before i tend to use the cincinnati graph data repo which is kind of low in the weeds if you will this is the kind of official way that they publish those so you may want to subscribe to those if you don't already um i know i think as employees we get subscribed to all of the alerts all of the time so they get routed to devnull for me but um so this is a great way to keep updated with what's going on there and then they do eventually many of these make their way into the release notes so if we go back to the docs here and release notes and then you got to come like all the way down to the bottom you can see all the way down here it talks about all the things that have been going on inside of there so to camilla's point the release notes are a phenomenal way to understand what's going on like you can see here i think we might have talked about this on stream i don't remember but like the minimum storage requirement decreased from 120 gigabytes to 100 gigabytes i can avoid a lot of things right it's better check the document before i go there then you have the great change to see the things like and come on surprise yeah and so we talk about deprecations that of kind of the non-api variety like we're talking about api contracts here but like even when like certain features or technologies um we don't see a future for them in openshift we will tell you that ahead of time so that you can plan accordingly and there's a whole section for that as well um as well as then all of our our tech previews and other things for what's new and what do you want to try out all of that is in here yeah so i did uh just to illustrate that in the release notes there's this deprecated and remove features and you can see like there's this very nice table that shows you all of the deprecated features when they'll be like cluster loader we can see is gaga deprecated uh which means that at some point it will be removed um so yeah release notes yeah release notes are they're long and boring and and oftentimes they're great just before bedtime reading but there's a ton of great information inside of there and i'll mention that the v1 beta 1 crds is actually in there and you can see it's been deprecated this is one of the apis we just talked about it's been deprecated since before six so uh or one a few rows up is i mean there's a lot of yeah there's apis yeah docker registry v1 v1 beta1 crds yeah there's a few of them in here um jumping back to today's topic rob do you happen to know if there is insights rules for api deprecations um i believe there are um i don't know if they're gonna wait to go live until like you really really need to do something about it like i don't know how much buffer room we give you there so i don't know if they're live right now but they do exist and may either go live if they're not uh closer to the four nine uh release okay we should we should ask john spinks about that chris yeah we should um just catching up on chat here so i i see an apologies for butchering any names um purkinje uh asking about using submariner so if you are deploying or if you're using acm advanced cluster manager uh and you have your clusters connected into that you create a cluster pool and you click the button that says deploy submariner and it does it all for you it's super neat and thank you rhope9 for linking the documentation there um i think i can take this one can you explain what deprecation actually means it means at a point in the future this feature will go away um and that feature could be right like the beta version of the api being bumped up to stable or like an alpha version just not getting adoption and fading out um that's kind of how that works yeah you know the official kubernetes definition of deprecation i can look that up but i mean that's the layman's term stuff yeah and it's it's kind of communicating an intent in the future so you're saying right this is deprecated now so probably don't invest in it or upgrade whatever and then it will be removed at some point and so the reason we're here today is because we've reached that point this stuff is actually getting removed right how how often it seems like 1.22 is kind of a big mark for this like how often do they actually get removed it's like in general it's not very often right like i think everything just came to a head in this release because of all the you know going to stable things yeah i think we there was like a if these things either don't progress to stable by x time period they need to be removed or just dropping a bunch of cruft which i think is mostly what this is is like look these things have been in here for years at this point like let's just finally get rid of them but on that um one of the links that you posted about the removals as you can see the things that are deprecated in the versions that they're going to be removed so i think there's some other stuff coming up in 124 that will get removed um so yeah but i think it's really only three kind of major releases that they have removed a ton of stuff most of it happened on this one and normally it's like this gradual process api we moved in 116 and uh 122 version of kubernetes and 125. yeah the release team is always good the kubernetes release team as well as ours um is always good at saying hey it's going to be deprecated in this release so you can actually like almost put a calendar entry in and say like if it's not fixed by then problems right i think one thing nice is it's likely to admit right they let they know it's like if they have a lot of operators installed in the cluster were they oh everything is working now but if the epigraphy 449 for example where this apis doesn't check cc anymore and they have the operators versus installation that they are using these apis then this operation will not work the important thing is that you need to ensure that all versions installed are compatible with the next version of the open shift before the dope agreements to avoid the ping and many hours of work afterwards to you know try to solve the problems exactly yep yeah so christian brought up a good point in chat which is um an api being deprecated does not mean that the feature is being removed and going back to that page or that chart that i was just showing we do show when features are deprecated and eventually removed as well but the api being deprecated just means that you particularly you know assuming the feature isn't also being removed just the api being deprecated usually means that you need to just move to the new version right which isn't always hard yeah usually when something is really widely used like the crds it's just a formality yeah it's yeah literally a one-line change often well we are we are at the top of the hour i know we don't have a hard stop today so folks if you have any other questions anything that you want to ask us please feel free to get those in via chat thank you rob really appreciate you coming on thank you for all of the great information today i know this is always a great stream when i learned something which happens pretty much every time so they're always great in my mind you know nice thank you to uh camilla and thank you to frederick i appreciate you both joining very much um i know that we we uh this one came together a little bit last minute um and you all were phenomenal and being flexible and accommodating so thank you so much um yeah we really appreciate it really thank you for having us here that was really amazing i really like it i learned too much today was really cool yeah so for our audience um please don't hesitate to reach out at any point in time if you have any questions about today's topic if you have any questions about really anything else related to openshift and and uh using openshift so you're welcome to contact me on social media at practicalandrew i'll throw chris under the bus at chris short on twitter uh you can also reach us via email we post our email addresses in chat but just reiterate uh i am andrew.sullivan at redhat.com and chris is short rehadhat.com if we can't find the answers we are always happy to track down and harass the right people so don't don't be afraid to ask us about anything and everything and with that being said we will see you in two weeks uh remember next week is ansible fest so be sure to join those um again sign up for brain dates if you want to have one-on-one with smes around many things not just openshift uh around ansible awesome thank you everybody are you attend thank you folks stay safe out there folks we'll see you soon you

Channel: OpenShift

Views: 365

Rating: 5 out of 5

Keywords:

Id: lQoXfXj6wDs

Length: 65min 55sec (3955 seconds)

Published: Wed Sep 22 2021