Hands on with EKS Networking (2023) | Amazon EKS Workshop

all right hello and welcome to another episode of containers from the couch today is a part two of a series we've kicked off going through the allnew eks workshop uh you can always find eks workshop at very simply eks workshop.com today I'm joined by my guest uh AWS principal developer Advocate sheel josi sheel how are you doing today I'm good uh super excited to be back U on the couch I guess um let's kick it off awesome let's do it okay so again I'm your host s Venom sheel is actually my teammate and uh a subject matter expert on all things networking with Amazon eks uh with this all new workshop which we Rewritten and revamped in 2023 we're addressing common and popular use cases that Amazon eks customers and just kubernetes users in general uh tend to run into so challenges and requirements today we're going to be FOC focusing on some networking use cases she's got an ex an existing environment of the Amazon eks Workshop up and running by the way folks if you want to follow along at home um you can always deploy your own version of the eks workshop uh environment going to the in your account setup steps following the steps there of course there's going to be a charge associated with running the eks cluster and some of the necessary dependencies there um but you can follow along at home or watch this video to watch an expert sheel do it please if you have any questions throughout our episode today feel free to drop a comment or if you're watching this uh recorded after the fact feel free to drop a question in chat and we'll get back to you okay so uh sheel now that we've gotten the kind of uh prerequisites out of the way I think we want to jump right into it I know we've got a very demo heavy presentation today of course we're going to be running through the workshop uh let's start with what the first module in the eks workshop is going to be and I think you've got some precursor information for us as well right yeah uh thank you s um today uh we'll be covering everything about eks networking right and as we all know Amazon VPC cni is the heart of it I thought it's better to talk about some of the kubernetes networking primer before we Kickstart the workshop and understand what cni is and how VPC cni behaves right um I'm sure like most of you joining already have some kind of kubernetes uh experience and know about cni but uh anyhow we'll just go through some of the basics here so kuber is networking tenets as we all know it mandates that all containers should be able to communicate with other containers without any super of the complex s being involved and also the IP address that the container sees itself is the same IP address that others see as well right so how does kubernetes solve this networking problem um something called as container networking interface and networking interface is a specification for kubernetes on how Parts get an IP how ports communicate with each other right and cni also assigned a network name space and a network interface to the part at the beginning of the part startup and it assigns an IP address to a pod and it also mandates that clean up cleans up the network names space when the pot terminates without leaving any residue behind and how Amazon eka solves this pod networking problem is using the Amazon vpcc cni um we architected bpc cni in 2017 um and it integrates natively with the Amazon VPC one of the battle tested soft defined networking uh on the earth and there is no overlay networking involved right and it also provides subsc latency for the port startup as well as the port communication so so sheel I just want to make a quick comment here that integration between kubernetes and the VPC uh capability in Amazon means when we start working with kubernetes uh today and we start doing things like creating services and um exposing CA uh applications a lot of the times there's actually a step here and you say oh let's go to the console uh and see the actual changes in the VPC I think that's really interesting where no matter you know what you're working with within kubernetes it's tied directly to those VPC Concepts um and and that's something we're going to see today as we start doing things with the cluster uh we we'll be able to see that it's actually driving um behind the scenes changes in the Amazon VPC capability exactly and um I think I'll just also go through some of the details and it will kind of make it clear right so what exactly those calls are and how does it make an API call and how does it all work uh so just like whenever you try to schedule a pod you use a kubernetes API right and kubernetes API hands it over to the Schuler and Schuler actually says okay this node is free let's go ahead and deploy or run this part on this node it invokes a cuet basically it invokes the cni add or the delete commands for the parts if you are basically scheduling a part then it's going to invoke an add command Amazon VPC cni as you see here has two components right a cni binary as well as ipam the plugin so what actually both of these things do um so you can provide the cni plug-in to a cuet using the network plug-in command and you can also say this is where my configuration file lies for a cni b and when a pod lines on a node the API server uh the cuet and the cni plug-in components work together to assign an IP address to the Pod sets up the Pod network name space and also the host name space using the V pair that we will just see in the next slide then also updates the route tables on the Node itself so what does this iamd plugin here as Sai just mentioned basically it integrates native into the VPC uh that also means we are assigning an IP address from the VPC side to a part and that's what iamd is taking care of talking to the VPC and the E to control plane API is making sure there are enough IP address so that these VPC IPS can be assigned to a pod we've actually got a question in the chat already from tar I'm actually curious to know the answer myself uh does does pkss use pause containers for pod networking and IP so essentially I'm guessing to to be able to reserve the the IP addresses um like a secret container to be able to to to to reserve them uh no that's what basically in ipam d uh which is AWS node demon said that if you are seeing on a note that does the job for you right um it's always running it sees how many IPS are available to be assigned if not it makes a call to the control plan apis to reserve those pool for you right so that's what it says here like cni plugin attaches Ani to the worker note um and also there are settings called as warm pool settings which typically we call it as War Target or the warm IP Target or the minimum I IP Target that you can configure right so what iamd is doing is basically it looks up at those values and then it assign finds a warm pool of secondary IP addresses or maintains the warm pool of the secondary IP addresses right and when a pod actually lands on a worker node it is assigned that IP address from the pool it's constantly running and if it sees that pool is depleting it makes sure it keeps an additional eni and also the number of the IPS um that can be allocated for that eni hope that kind of makes it clear yeah that makes sense yeah and as in when the IP addresses on e z are exhausting right they are assigned from an E1 and so on and also because what um a cni does is here it assigns the IPS to the pots and also assigns those IPS back onto a node as a secondary IPS and that makes it number of parts and a worker node to be limited by the number of the IP addresses and the number of theis that a node can handle or an instance type can handle so whenever you are actually selecting um the instance type for your noes please be uh aware of the number of the in Enis and also the IP limits for those Enis so the next is prefix delegation as we just saw when you actually schedule a pod a secondary IP is drawn from the pool and that IP is assigned to a pod and this means you are limited by the number of the secondary IPS that can be assigned to Ani and this is where a cool feature which is called as prefix delegation comes into handy um and in here what happens is in instead of just having a secondary independent IP you can assign a sl28 prefix and the sl28 prefix is a constant regardless of the instance type you can just assign sl28 prefix uh the first IP on Ani is reserved for the primary IP and all the other 15 IPS are for the P pod IP assignment on that prefix and when a pod actually lands on a worker node it is assigned an IP address from within that prefix so just in case like as an example if an instance type support three I secondary IPS in here you should be able to assign three prefixes which tremendously increases the number of the Cs that you can be running on a on any note so so she we've got a couple of questions coming in and I want to bring those up while you have that chart up so from cloud Surfer 22 as a best practice what should be the CER ranges for a subnet of course I think we're talking about a subnet uh in VPC if I understand the these subnets need to have at least uh six IP addresses that eks will will use itself um so I think the recommendation is at least 16 but there we're seeing sl24 in in your subnet correct yeah uh so there are like you know six IPS that you're required because eks uses something called as cross count I to be able to communicate with the kubernetes data plane I the worker notes that are running um in your account and slash6 and in the example that I am uh using it's a sl24 um it all depends on the scale um and the number of the paths that you're going to be running right um so please make sure you kind of understand the part density and size your subnets accordingly got it okay so when similar to the secondary IPS independent IPS here when prefixes on E zero are exhausted they are assigned from e one the iamd that is running assigns an additional eii to a worker node in the back end and then make sure it also keeps in um the prefixes in the pool and that value you can control using the warm prefix Target as well so let's go ahead and jump into um the workshop itself so we have at least I have provision and environment here right um and what I'm sharing as part of that the workshop is the workshop infrastructure already set up a clusters for you it's a single cluster right sayi if I'm not wrong we just uh spin up a single cluster at this point of time that's right that's right a single cluster a few node groups in there but really just one that we're going to be working with as you can see here the three node groups that that that are populated uh in the console um for folks that are watching at home and want to follow along of course you can follow along with us here today or spin up your own copy of this uh environment we have terraform based instructions that allow you to spin up this entire uh environment in your own account um and of course if uh you're you're working with um your favorite field uh seller from AWS maybe they're running this event for you um you you would get access to an environment as well for example if you're running a workshop at an AWS event like reinvent or a customer event uh you would also get access to an environment like this yeah so I have like I have the cluster running uh the node group already provisioned so let's go ahead and look at the vpcc setting here uh for the purpose of the efficiency uh when you spin up this Workshop uh the bpcc is already running in a prefix delegation mode so I'm just clicking on one of the instance uh here one of the worker node um as you can see there are twois that are configured on that node and already two of the prefixes configured so why is that so s when I bring uh while I'm bringing up the console can you talk about the sample application um that Workshop uses yeah definitely so uh obviously AWS Amazon web services we've got to make our own e-commerce sample application uh I'll tell you folks if you want to learn more about the sample app even outside of the context of just eks uh We've we've developed this app to really be a demonstration of container capabilities um so I'll drop a link to it in the chat here um but you can deploy this as containers on app runner on Amazon eks I believe we've also added support um for things like docer compose uh and really it's just a microservices based application um it demonstrates a number of capabilities including stateful stateless applications uh as well as message CU Services uh at the end of the day just a simple retail e-commerce sample app uh made up of a number of different microservice components and you can see those here by listing the pods um well in addition to all of the different services that we have running in the cluster uh the the pods that are making up the application itself yeah and we also have some of the bare minimal addon that are required or which can I think help us understand the application a little bit more so have as you can see here there are a lot of um applications that are running the poths are already running and that's where when I went to the instances here you can see um basically uh the network being configured and eni being configured and prefixes are already assigned so let's go ahead and look at the internals of the vpcc itself so what I'm going to do here is I'm just going to uh describe um the AWS node it says yeah the cni is running and 1126 is the version of the cni and let's see uh the settings of the VPC cni itself yeah as question in chat Cho I think maybe we need to step back just a second what is delegation mode so of course you mentioned that the the cluster already has prefix delegation enabled um and I just want to say we're about to answer this question going through the workshop itself but maybe it's worth talking about uh really quickly sheel what it means to have a cluster with prefix delegation enabled I know we did it already in this eks cluster but can we quickly talk about how one would enable that yeah uh so as we explained before actually the parts reive an IP address from the VP cider and then those IPS are also assigned as a secondary IPS on a worker node right uh that kind of limits um to ability to run more of more poths on a given note and you might see your resources especially your worker notes are underutilized so VPC uh launched this cool feature called we call it as prefix delegation and also you will see in some of for dogs prefix assignment right instead of a single IP address you can now assign a slash 28 prefixes uh to the Enis right if a eni supports three secondary IPS now in case of three individual secondary IPS you can assign three prefixes of sl28 size which means instead of one single IP There are 16 more IPS that you can uh draw your IPS from so that tremendously increases your pod density on a given worker node um if it was 19 uh pods on a given worker node with the prefix delegation that number might increase up to 110 on a smaller instances and on some of the larger instances now you can run pots up to 220 as well so so basically depending on the type of node you have sheel you're limited to the number of IP addresses that are available that limits you for the number of PODS now of course customers might want to deploy many smaller pods on a smaller instance and to enable that increased pod density uh you can enable prefix delegation to essentially give uh the ability for more of your pods to be assigned an IP address and then maximize your resource utilization on that given note right and that's exactly I think what what we're going to be seeing today with this first uh lab exactly yeah and then the way that you enable prefix delegation is using this flag as we said we enable prefix delegation uh on the workshop as you spin up the workshop the prefix delegation is automatically turned on uh it's the workshop infrastructure that is enabling the prefix delegation but if you are actually running your like creating your own cluster the VPC cni still runs in a secondary IP mode and you can follow the steps in our user guide and also the steps mentioned here on the uh Workshop to enable the prefix delegation so what what we've done there is double check that the prefix delegation is enabled um okay and and now in in this step um well I I'll let you explain it yeah so this is this is same thing right I I just explained it through the console uh I showed the number of the prefixes and we are just using the AWS um command line here to show for each of the instances how many prefixes have been assigned and and for our folks watching at home essentially what this uh cider range is denoting here is that uh we've got the sl28 so the 28 the four bits here uh will lead to 16 additional IP addresses and you can somewhat kind of see here for example in that very bottom one that's 104210 6428 so the one right below that one right sheel so we added 16 to that one that actually gets us to next one that's right below it the the 104210 180 so you can see how that sl28 corresponds to 16 reserved IP addresses um and and those two together make 32 and the three for that instance will make obviously um 48 uh so that's that's kind of the number that we have Associated here yeah and you see on this instance there are three IP uh prefixes and maybe on the other instances there are two that's just because there are more parts that are being you know scheduled on this note got it so that's dynamic as we schedule more pods more prefixes get created exactly so let's see here so what I'm going to do uh towards the same question right I'm just going to run one of the PA container here right it's just we will spin up 150 parts and see how prefix delegation actually works um in action okay so 150 pods if if we didn't have prefixed delegation enabled we would clearly run into a limit here um that's yeah so maybe I'm not in the wrong oh maybe I'm not in the right main space is not oh I I think I know what's going on here shito I think you just need to run the the reset environment command because the first time you uh access this environment uh it doesn't actually deploy the sample app um and and the name spaces so for for folks that are watching at home um the very first time you spin up the eks cluster environment the first step in the introduction is actually guiding you through deploying that sample app that we just talked about um but we've made the lab modular so if you want to just skip the intro you're an expert you want to jump right into the networking module we created this handy reset environment command which enables you to get that environment up and running ready to go um so that you can actually jump right into the prefix delegation step uh she I'm guessing you usually work with a cluster that's already run this command before but working with the new environment you do have to run that yeah and each of our page actually describes that you know um we recommend you to run this reset environment but if you are just going from start like you know beginning to end then it's all fine uh but if you're just jumping across different sections please run reset environment don't make a mistake that I me just uh well we wait for that to get ready we've got another question in the chat is IPv6 supported um and so I want to say yes but with some caveats right and and and she I'll let you elaborate yeah uh yes we do support IPv6 today and IPv6 is supported in a prefix uh mode by default however we have not added IPv6 section to the workshop as of yet it's um coming soon um is I I think we just wanted to keep uh simple for the initial part of the workshop so that everybody understands different concepts of eks networking and go in detail of the vpcc and the different mode that bpc cni supports but of course uh we do support IPv6 and we do support IPv6 in prefix delegation mode excellent excellent all right while we I think the app should come up any second now i' I've never had this Comm in you know take longer than a couple minutes oh there we go uh now we should be able to go to that first step there um yeah let me go to that section uh let me go ahead and try apply yeah cool so what we're doing now is just deploying 150 copies of a very simple pause container right and and uh kubernetes will anytime you spin up a pod it's going to assign an IP address to that that pod um now of course these are cluster IPS so the IPS are not accessible externally it's just a way within if you're within the kubernetes network to be able to uh for the pods to be able to communicate with one another um so I'm guessing this should happen fairly quickly uh 150 pause pods um but what's probably more interesting here sheel is while this is happening can we run that uh command again to see the ipv4 prefixes associated with every instance ID I think all of the pods are up and running to let's just go ahead and see if we have yeah uh they are all up now we should see more prefixes yeah okay as you can see here some of those SPS that we just ran landed on to the instance where it has only two prefixes before now you see additional five prefixes being assigned right and and for each node as well because those pods are being distributed across the three nodes that we have that is correct yeah okay excellent so it it it looks like it's working as expected we've got prefix uh delegation enable which means now we have access to many more uh IP addresses for pods to be assigned if we didn't have enough what's the error that people can expect to see she do do you happen to know that off the top of your head yeah I I mean parts will be waiting to create it because there are no IPS available uh to be assigned uh to a pod fair enough yeah so we just have unschedulable pods that makes sense yeah and also there might be another scenario where we reach the max spots uh limit on a worker node and then your all of the parts might not be scheduled either got it um and you know I'm guessing with IPv6 like you mentioned it's automatically enabled and of course with IPv6 you you just have access to way more IPS uh that that pod density issue is less of a challenge right that is correct yeah uh we got we've got a good question come up actually again from cloud Surfer so as pods terminate are those IPS reused uh does it try to assign IPS from contiguous blocks so start with the first one okay so what happens yes when you basically delete the parts those IPS are given back to to the pool and depending on the warm settings that you have um it those IPS might not be immediately released back to the ec2 pool right we try to M maintain the warm settings and if those setting limits have reached then the IPS will be released back to the ec2 uh pools and yes that's a good question actually uh The Continuous block of the IPS uh if your subnet is highly fragmented you might see an error when cni tries to basically draw the sl28 prefixes from the subnet uh then you will see a failure and it will not work so you will have to make sure that you have the continuous block of the IPS available and also we recommend that you reserve the prefixes in case if you see on that these blocks are not like IP blocks are not continuous yeah yep yep and and really we can just see here from those prefixes that it looks like it tries its best to assign the the contiguous blocks when possible cut yeah yeah okay so is is that uh I think that's all that we've got for prefix delegation right I mean again just to kind of recap this is a a capability to enable um customers who are running into IP exhaustion limits you know pods that are unschedulable because they they they don't have IPS that can be assigned um this means even with smaller nodes which don't have as many Enis attached you can still deploy a larger number of PODS yes yeah before I move on to the next uh setting of The VPC just going to make sure I run this while it is doing the work uh in the background let me go ahead and go talk about the custom networking uh I there's a step in custom networking we have to provision a new node group right so yeah maybe that's a good idea so let me go ahead and do that as well yeah yeah I I figure that's that's uh that might be a good thing to to just get that node up and running um I think it's in the third step the provision a node uh a new node group steps yeah um so just we're just doing some magic behind the scenes here folks if you're running through this at home of course go step by step um I think one more step sheel uh and then you'll see it I unless yeah one more there we go um one of the steps in this uh module we have to create a node group creating a node group takes a variable amount of time uh I'd say about five minutes or or more so we want to just go ahead and and and run that um in the meantime um yeah and we will explain why is it really required uh when we get to the step of the workshop just wait here until this reset environment uh come on completes okay great so we've got a a question while you run that uh Sho okay a got to creating does prefix mode prefix assignment mode prefix delegation mode whatever whatever you want to call it does it add more uh eth either end points to the instance um so thinking the answer is no but I'll let you elaborate yeah so if you look that in the contrary maybe you might see a fewer Enis uh than uh in the secondary IP mode because you are able to schedule more pots right and maybe just one prefix might be just enough for the instance type that you are on and the number of the pots that you are scheduling so it's not assigning more Enis probably reducing the number of the Enis on the worker node that you that what what does the eth stands for like so the you in the diagram you have ethi so the elastic network interfaces what what is what is the is that ethernet yeah where do you see which diagram I think it was in one of your previous diagrams um yeah the E this is e zero and the e one right that's how you will see on the worker node uh but these are the elastic network interfaces that are assigned to assigned to the node itself right that's right all right so we're on to custom netw working now um we we've got the the the prerequisites spinning up in the background um what do we got here yeah so we saw how vpcc works and what secondary IP mode is and the prefix delegation and with the prefix delegation you kind of solve the Pod density issues and typically pod networking the VPC cni mode um like the VPC has a primary cider and also node and pod IPS are assigned from the same address space and in most of the scenarios what we hear from the customers they have been using the vpcs that have been um like you know designed before the kubernetes era or also they did not know how VPC cni works or probably they might be using a different cni and they have actually created a smaller subnets right um and there are the cider is full there is there is no more space for assigning more IPS from that cider and that's where custom networking comes in custom Network again vpcc uh the main tenet provide all of or integrate with all of the cool features that Amazon VPC supports right and one of that is also able to assign the secondary cider and that way you can extend your IP availability to the cluster so what happens is you can go ahead and attach a secondary cider uh to your VPC and this secondary cider is typically from the CG space like 164 space and when you assign the secondary cider you will follow all of the best practices that are laid out by the Amazon VPC itself and eks kind of just follows or piggy back on the Amazon VPC best practices for assigning the secondary Siders you will set up the subnets with the secondary Siders as you you can see here uh I have created two uh private subnets I think part of this exercise we are using four subnets but we will see when we get there and here the worker nodes receive an IP the primary IP of the worker node itself is from the primary cider and the pots actually receive an IP from the secondary cider and the way that it works is just like other configuration or the parameters that we pcni I supports you can actually enable the custom networking using the custom Network flag and vpcc ni supports a custom resource definition called as eni config and this is where you will say this is the subnet that I'm going to be using for a particular availability Zone and when you specify that uh and when you try to schedule a pod the config that is applied to that Zone and on that node um will be used and your paths will be scheduled on the secondary side or subnet so let's go ahead and see it in action so this is one of the sample eni config so by default if you are not using uh your custom label the zone is used uh like what you can see here is this config is named as uswest 2A and also the security group that you want to be assigning to that eii and the subnet that this particular eii config uh is going to be uh using to assign an IP to the PS so I have a a question here Sho um you know we talked about prefix delegation now we're talking about custom networking um you talk about the use cases where one might want to use prefix delegation versus custom networking uh for for the IP addresses issues yeah uh so custom networking is typically when you are seeing an IP exhaustion right like your cider like you have used up all of the addresses available addresses in your primary cider and you want to basically assign a secondary cider so that you can schedule more more pods prefix delegation is you still have IPS available in the primary cider right and you want to increase the density of the pods that means you want to run more pods in a given instance that's when you will use a prefix delegation however you can use prefix delegation along with the custom networking so what does that mean is whenever you have enabled C prefix delegation along with the custom networking um basically you are applying the prefix delegation on the secondary cider right as the pots receive an IP from the secondary cider um you can also actually apply the prefix delegation instead of the secondary IP um you are basically assigning a prefix from the secondary cider thus giving you best of both the words you are also increasing the part density on a given instance um or a worker node along with that you're able to scale beyond your IP limitations gotcha and so combined you know that that's something that um is very much possible uh to further increase the number of Ip exactly yeah awesome so what what I'm trying to do here is just trying to see what are the ciders that are actually uh applied to a VPC right so as part of the workshop infrastructure creation we have already created a secondary cider um here as you can see 16416 has already been assigned to this VPC and even when you are running this workshop on your own uh when you spin up a workshop you will see both the Siders have been configured we are not using the secondary cider yet we will see how we can apply that uh or how we can leverage the secondary cider in a bit so let's go ahead and describe the subnets and this yeah uh you will see we have created subnets both in the primary cider as well as the secondary cider in each of the availability zones that are available for uh the cluster so this is how it's going to look like um as you can see there are three private subnets in the primary cider and also three uh secondary subnets that have been created across three availability zones so let's see what are those subnet yeah these are the subnet IDs and let's go ahead and set the custom networking config flag to true and the C config caml is just describing um you know uh the subnets that we just assigned and these A's are mapped to the availability Zone that we just shared in the previous uh image so let's go ahead and apply these eni configs to the cluster yeah so the ni configs are available and let's examine when we actually have uh uh applied the configuration it also requires that we have to enable the label by default it is mapped to the zone and also in this example we are just using the Zone uh availability Zone labels if you have a different requirement um and if you want to use multiple if you have multiple subnets in the availability Zone um just using the availability Zone level is not going to be sufficient so you might want to create a unique label to be used um so that poths are are scheduled on the different secondary subnets that are available within that zone and here we already performed this step like actually creating a node group and why is this important custom networking is not going to be enabled by default on the existing nodes the parts that are running on an existing noes will continue to be running with the IPS from the primary cider right and whenever you are seeing an IP exhaustion issues or whenever you you you want to basically deploy these pots onto the secondary cider we recommend you recycle the existing nodes if you think that you're fine with the existing applications running in the primary cider you can leave that um uh node group and the PA running on that node group and then create the new node group for the new applications that you want to deploy in the secondary CER but we recommend that you recycle the nodes uh for the parts to be uh getting an IP from the secondary CER so let's go ahead and see that if that node group was created fine okay it is created it got cre so so I had a quick question so you set the Damon set on AWS node telling it to enable uh the the custom networking capability now um any pods that we've already deployed are not deployed on this new node yet so they're G to be the the 10 dot instead of the 100 dot I ranges um now I guess my question is do you need to explicitly define whether the pods are deployed on one or another or can you just let kubernetes handle it um and and it'll automatically you know spread the load of the of the pods across uh you know custom networking enabled and non- enabled nodes uh that is completely fine if you think you have uh IPS available in the primary uh cider then you can do that right and kubernetes yeah to do the magic for you but if you think you're already running into an IP exhaustion issues in the existing cider then just doing that is not going to help you I see I see and and I think another way to phrase this is a question that just came in chat so with custom networking enabled uh which cider takes precedence for new pods and deployments and then for the existing pods and deployments as we just discussed uh they they won't be rescheduled until at least you restart them or you tell them you know with the node selector like we're about to do to to be to the new one so at least for in this example what we have done is we have actually labeled the node uh as custom networking right I am telling declaratively telling okay I want this check out applications to be deployed on a new node with a custom networking tag so that it receives an IP from the secondary cider right otherwise it's you know luck of the draw it could go on the first one on the first three nodes or the new one that we deployed depending on which one it gets deployed we would get either one of those 10 dot IP addresses or the 100 dot in the secondary IP ranges correct yeah okay let's go ahead and run this sample app so what we're doing here is we're customizing the checkout app so the deployment is already there but now we're saying let's update that deployment and add this node selector to do uh to to tell it to deploy the checkout deployment pod on the custom networking node so it's says looking for any node with that label enabled on it yeah and another thing that to note here right uh so everything is VPC fabric when you create a new cider no additional changes are required at a networking level right the VPC fabric is going to be taking care of basically the networking across um or the communication across these different parts that's right yeah so the sample application the checkout is now running as you can see only checkout app has or a checkout pod has an IP from the secondary cider versus the all the backend parts that this checkout depends on still runs um on a primary side or on an old node that's right so when we created that deployment kubernetes realized hey the config changed we need to restart this podt um and then that's what enabled it to to work got it yeah MH so how do we clean up I think that's a critical thing too and just go ahead and reverse all of the setup I'm just telling disable the custom networking by just setting it up to false and I'm just going to go ahead and delete the node group additional node group that I have created and I'm going to also revert back my checkout app um to be running on a primary cider yeah just applying that custom it's definitely valuable to see how this works to to see the cleanup uh but folks that are following along at home I think one of the things we've done sheel is also program this into the reset environment Command right so anytime the user runs reset environment um you know say you got halfway through the lab and then you decided hey I want to go to the next Lab just run reset environment and we'll get the environment back to a regular state so you can be kind of sure that further modules will continue to work as expected as well any questions um on the custom networking that you're seeing s well um I think we've addressed most of the questions on custom networking here so far uh I think let's see um for folks that are watching reading along uh if you have any other questions for for how some of these things are working please drop us a question here uh we're resetting the environment before we move on to the final uh module in the networking piece which is going to be security groups for pods yeah and another thing to actually note here right when you are basically using the custom networking the primary 80 is going to be getting an IP from the primary cider so technically one of thei is kind of not not used with the secondary cider here so your pod density is going to be affected slightly yeah when using the custom Network networking there is one lessi available for the Pod IP allocation so pod density when compared to the regular pod networking um is less when using the custom networking right we do have a question coming in here she so from tar how does cni play into eks fargate clusters uh we don't have to assign prefix delegation or custom networking and fargate right uh no uh however basically the SEC next option that we are going to be showcasing um you will see uh how that is extensible to farget as well yeah and and T I know you had an earlier question as well about third party cni plugins that are also available so one thing I just want to clarify here is that the VPC cni plugin is the only one that's officially supported by Amazon eks uh but you can work with other compatible cni plugins um you know is surveillance psyllium or you know Calico and a number of others I think our recommendation is that you should get support from them if you want to use those cnis uh and of course here being Upstream um conformant version of kubernetes in Amazon eks you can use any cni of your choice um but we do recommend and support Amazon VPC cni plugin yeah so fargate is just like its own independent task or a unit right so only supports VPC cni uh if you're if you're running in fargate okay so let me see if uh uh yeah so what's happening here the node group is deleted I'm just going to go ahead and apply uh the checkout patch yeah and then also run the reset environment while we go through some of the basics of the security groups so so security groups right um each worker node has a primary ni and uh one or more secondary IIs that we just saw pots are assigned an IP uh from the secondary IP addresses and in this setup worker node and all of its spots basically share the same security group that is attached to the primary of the node there are many cases wherein you want to use the security groups for a part separately and not want to use the security group that is assigned to the worker nodes right um I think one of such scenario you can think of is like you you are running a basically a backend part that requires an access to an RDS database and most of the organization ations have already built some of the compliance programs around the security groups and they like the AWS security features um that security groups provides them and want to and wanted to extend those similar features to the paths running on kuties right and they wanted to reuse all of the operational knowledge tooling and experience around the security group policies rather than implementing the kubernetes network policies that is one way to think about um and this is what exactly security groups for pots is offering you right with the security groups uh for parts what uh you can actually do is assign a separate Security Group for a pod and how does vpcc makes it happen you can of course enable uh it using a different setting called as enable pod eni so what happens when we do that there is a resource controller that is running VPC resource controller that is running on a control plane um and it creates a trunk eni and it attaches that trunk eni to the node and you can define a part Security Group Policy which is again a custom resource and when you actually apply the security group policies to a pod the VPC resource controller that is running on a control plane creates a branch eni and actually whatever security group that you have defined under the security group policies is assigned to the branch eii how this is a sample of the Security Group Policy so what we are doing here is it's like a sample python application and when you for the given selectors what I'm saying here is okay go ahead and apply this specific Security Group group if you are deploying a python app you would want to run it against the branch and then apply this specific Security Group uh as you will see here all of these trunk Enis and the number of the branch Enis for a given trunk eni interface is additive to the number of the parts like like the max parts that you can run on an ec2 instance um maybe there is a link uh that I can share um in a in some time where it shows know the number of trunk Y and the number of the Branch Y that you can be creating per E2 instance type yeah and worker node gets a label which says okay has trunk attached and only when that label is available the resource controller knows about it and the trunk is created the poths are basically scheduled on that node and the branch is created in the security groups are attached to that got it and and when you enable this mode she uh to to use the security groups per pod um do you do all the pods that don't specify a particular Security Group Policy uh do they just default to the the standard they just default to basically uh The Standard Security Group which is assigned to the primary of the node got it and then for any pod that actually has that additional Security Group Policy a trunk and gets created again this this comes back to our use case of you know what what if we have a specific pod that we want to have have a different egress policy attached than a different than another pod right so one pod you want to restrict internal access and another pod to allow ex external um you know one approach we might say just put them on different nodes but of course there's always going to be a use case where we want to have pods to have you know fine grained access controls even when running in the same node that's really where this feature comes into play right exactly yeah and like you have front end we have back end and your API passs which are running like you know on the same node and you want to maximize uh the resource utilization of a particular node then this is a great feature to use excellent yeah so let's go ahead and uh uh so we are going to use the same catalog uh application here um and the catalog we have the my sequence also running within the cluster and what we are going to do as part of this exercise is going to be creating an external Amazon RDS service and allow only catalog application to be talking to the RDS service right so let's go ahead and see if the catalog pots are ready and running yes they are um and let's look at what is the database endpoint that it is pointing to right now so it's just pointing to the in-cluster my MySQL that is running we are going to change this catalog application to connect to the Amazon RDS instance now I like that with this module we're we're killing two birds with one stone we're we're showing how folks can integrate and migrate a built-in database to a manag service while showing the best practice of uh the the networking considerations using security groups rep pods that's that's really interesting yeah yeah so as you can see just the DB end points are again uh this is a config map basically right where we are storing in all the secrets and all the also the DB end points and here it's still pointing to the catalog MySQL and we we will just go ahead we have created already RDS database here so let's see if that RDS database is available in this environment yeah uh so it's basically eks Workshop catalog uh endpoint and I'm just going to go ahead and apply the change uh here so it's just again a customize that we are going to be applying and changing the endpoints to use the catalog RDS right so all that is done applied the change it it seems good let's go ahead and see okay yeah watching your what we what we're applying here are um yaml changes using customize customize is you know native kubernetes configuration management it's actually integrated directly into Cube cuddle um and you can apply using dask the reason we use customize in the workshop is to make it really straightforward to see exactly what is changing Shea if you don't mind scrolling back up to that customize and going to the diff tab in the actual instructions if you go to the diff tab it'll actually oh I guess we didn't use that here but regardless point being um you'll notice a couple will oh there you go there it is the if you hit diff over there um you can see exactly what we've changed so the patch itself could be kind of long and complex uh this clearly shows you all we're really doing there with this particular one is changing which end point the catalog uh application is using not the built-in MySQL but the RDS that we've already as part of this environment yeah so I went ahead and applied um the patch and what we can see here is yes the database endpoint is now changed to the RDS so let's uh uh okay so what we want to do next is okay is it applied properly okay yeah that's good we we would want to recycle this for right because to be able to connect to the RDS let's go ahead and recycle it so that the new config map changes are pulled in let's wait for few seconds for the changes to be applied so uh the security group um basically that's going to be used by default is is going to be the one that is assigned to the primary ni and that doesn't have actually any access enabled to the RDS endpoint and we can confirm that here using okay when I in fact we expected that timeout to happen here right because using a default Security Group in fact if we went into the Amazon console and the VPC and we went down to the security group for this particular uh subnet then what we would see is that there's no outbound rules set that basically means there's there's no way for any pods within this uh Security Group on this node to even access a service outside of the the cluster and so obviously it's going to fail accessing RDS yeah so so that's what is unable to connect because the E to Security Group that's been applied to the RDS datab database doesn't allow you to connect so what we are going to now do is uh go ahead and apply a security group and the rule to enable access to the RDS database um so let's see what are the different security groups that are applied to yeah okay so we have already created the catalog Security Group um ID and then it just allows access to the RDS database as well so um yeah so that this is a security group policy that we were talking about earlier here what we are seeing is just if the Pod has a component service just enable the security group um for that catalog pod so let's go ahead and change that Okay so we've done there is essentially um we need to delete the Pod right so that it can pick up that new uh Security Group because this was not actually a change to um any config of the Pod itself but uh the the node that the Security Group Policy that's aligned to the node it's uh so for it to pick up that change we're manually deleting that pod to pick up U the new Security Group Policy so let's see if it is able to connect to the yeah now we see that we are able to successfully connect to the Amazon RDS database after applying the right security groups to the Pod I think yeah we do also have yeah and another way that you can actually confirm that it is using the branch and the security groups that is assigned to the branch why via The annotation um when a pod is created we also add annotation called as a pod and this is a specific y um that has been uh you know assigned to the Pod and I think I'm not sure if if the events are still there and not rolled over you can confirm that using the cube cutle events let me see if I can go to the console and if you can see there is a branch that is created um yeah so this network interface when I click it says the description as a branch basically and please remember like if you go to the network interface there is nothing like a trunk uh eni that you can create on your own these features are only available for AWS services such as ECS and dkss yeah um I don't think you can create a eni as a type trunk today I don't think that's made available yet uh we've got a question in chat and and uh I'm curious of this myself um well fair Fairly specific here do the VPC flow logs um actually in actually display the accept reject logs uh for when these where these Anis are set I think so yeah um I don't know whether did we enable the flu locks for this Workshop we should have probably gone and confirm that I I'm fairly certain we don't uh but that's that's interesting and um you know that's en here yeah but if they were enabled users would be able to see the you know the packets whether they're accepted or not coming from eks pods and IPS exactly at the end of the day it's all VPC Network yes right right good question and uh you know that that raises a good question we should probably make sure we have a dedicated module talking about VPC flow Lo because super valuable for uh you know keeping and auditing the traffic uh coming in and out of you know your VPC which is you know underpinning all your pods and applications in eks yeah and also I think did you share the eks best practi I on the uh yeah uh you know what I'll I'll I'll share it one more time because I think it's honestly such a great resource and we have a dedicated networking module in the eeks uh best practices guide which talks about everything we've talked about today and more in depth uh so I'll drop uh a link in chat here yeah and I think somebody kind of requested that we can add all the observability um practices to the networking best practices section yeah please stay tuned for updates such as VPC flow logs and metrics server details getting added to the networking section yeah yeah uh and we got confirmation from someone else in chat that the VPC flow log does show it so thank you poen um and and folks that are watching at home I know that you know sometimes even partners and folks with other external open source Community projects that are watching are curious how they can get pieces contributed to the eks workshop um so folks if you are interested in learning more about the eks workshop it's a completely public uh project we want to govern this in the community um I'll drop a link to the GitHub in chat here uh in fact we have our first eks Workshop Community call this Thursday if you're interested in suggesting enhancements uh getting involved with the workshop contributing yourself um please go to that Google doc that we have there um please feel free to add that to your uh calendar if you just want to leave us some feedback that's also fine as well uh we we welcome you know honestly any feedback uh and we we ask that you you know uh work with us in through PRS or issues or whatever it might be um as an example uh we have some pretty exciting stuff that we're working on uh in the networking module coming up here uh specifically uh VPC lattice which was announced very recently we have a module in the works uh in fact there's a PR open right now and if you're really curious you can actually go look through the pr uh and and and see it uh in its early stages as well but it's a great way to get involved in the project yeah I just wanted to show the trunk Andi um being created at an instance level right if you log into the console you should be able to see it oh there you go then all right she I think uh we've got pretty good coverage here I think we did all three of the modules that we have as part of the networking and we did it pretty quickly and you know uh I think in we we had an extra 20 minutes that we thought we might need here but um you know we're we're done a bit early is there anything that you wanted to to share with folks here uh I think this was a great uh session um I would just ask um people who are attending to leave us the feedback and also if they want to see more sections added to the workshop yeah definitely um folks I want to thank you for tuning in I know we couldn't get through all of the questions here uh I wanted to make sure that we stayed on track um we are going to continue to do these Workshop feature episodes in in fact if you forgot or or sorry if you missed the first time we did this going through the basics of eks Workshop uh be sure to check out uh the link that I dropped in chat um that was an episode that I did with Justin Garrison also from AWS going through the fundamentals uh of the workshop uh which is a different section uh today we covered networking which is a slightly more advanced topic upcoming we're going to have dedicated videos where we invite the smmes for automation observability security and more um so be sure to stay tuned for that subscribe to Containers containers from the couch on YouTube uh and and you won't miss any updates that we make there um sheel I want to thank you so much for joining me today for this excellent session and and going in so depth uh for for the actual um you know for the networking piece we had a question in chat actually asking is it possible to get access to your slides uh I don't know if you have them published but if you want to drop a link to me uh I can go ahead and add a link to the slides in the description of the YouTube uh I'll probably get that in later today um and with that chel I want to say one last time is there anything else you want to leave uh with folks uh today yeah I just wanted to highlight the sample application we have done a lot of work um you know and S and few of the other team members have spent a lot of time I think it kind of helps us to understand all of the aspects of the kubernetes so uh kudos to the entire eks Workshop team absolutely it it definitely takes a village here uh now that we're completely open source we'd love to get more engagement from the community um shidel again thank you so much for joining us all of the viewers at home thanks for the excellent questions uh we're probably going to take this video and add it directly to the eks workshop so for folks that want to follow along and not just read through the instructions uh they can they can do that as well um all right then with that I'll say Ad do to all of you thanks again for tuning in um and we'll see you all next time let's find the music where's the music here we [Music] go [Music]

Channel: Containers from the Couch

Views: 10,214

Keywords:

Id: EAZnXII9NTY

Length: 69min 27sec (4167 seconds)

Published: Tue Apr 11 2023