AWS CloudFormation Modules - A New Way to Reuse Templates?

[Music] welcome to cloudera node plus your source for exclusive videos and online events for aws professionals my name is michael vitik and today i'm going to introduce a new feature of the cloud formation service and this was actually asked for by the community so thank you very much thomas for asking this question and the feature i'm going to talk about will help you to reuse cloud automation templates or basically the question is will this feature really help you to reuse cloudformation templates and what other approaches exist so what's the feature i'm actually talking about it is called cloud formation modules so let's dive into the topic and see what you can expect from this video so first we will look into the motivation for this topic so how can we reuse templates why should we do that so that's one of the first aspects we're going to talk about then i will give you an introduction into cloud formation modules so how do they work what are the pre-requests for using them and what are the kind of goals behind this new feature i also created a demo where we set up a vpc using modules that i created and i will share the code with you so you can use that in your own projects or kind of use it as study material last but not least we will talk about the good and the bad parts as always we don't only look into what this feature is kind of good at we also look at all the limitations the pitfalls the problems so that's um the outline for today so let's get started so this is how a cloud formation template looks like so for example this contains a typical web application so to deploy a web application you actually need a vpc you need subnets and i mean the truth is you actually need a lot of more resources you need route tables you need associations you might need again not gateway you need access controllers so it's a whole list of resources only for the vpc we put in also our database into the same template so it's a database instance it's a subnet group i mean there are a bunch of other resources that might be beneficial like a security group and then we have the auto scaling group we need a launch template and we need the load balancer the target group the listener and all kinds of stuff all kind of added to the same template and this is i think the first approach this is also the approach that i took when i get it's when i was getting started with cloudformation the problem here really is um one week later you want to launch a web application that is slightly different than this one so how can you do it so you copy the whole template and then you have two templates problem is you make a change to one template you forget to um kind of fix that problem in the other template as well and now we have kind of chaos so how can we avoid that so one strategy is to split things up into multiple templates for example one good idea here could be to have a vpc template so you can split out all the vpc related resources put them into a single template and then you create and template for your application and this only contains the resources that are needed for the application so for example the database and the load balancer but now the problem is how do we get the vpc information into this template and the answer is parameters so for example we need the vpc id as a parameter we might need the subnets as a list of strings or list of ids and so we can kind of combine the two templates together but this requires manually wiring up those things for example in your deployment pipeline so the problem with this approach is that the list of parameters can get really long so i mean a vpc contains all kinds of things not only the vpc and the subnet ids you might need additional information uh in your application stack and then it grows and grows and grows so there is another approach that kind of solves this problem of lots of parameters so again we have a template for vpc but this time we create a so-called output and an output can be exported so an export has to be um it's basically key value pair information and the key has to be uniquely identifiable in your account region that's why usually you prefix the output with the stack name which you can access in cloud formation using the aws clone colon snack name and variable and then you kind of append the key for this information so for example i append the vpc id same works for subnets so now we look into the app template here we only need a single parameter which references the stack name of the vpc stack and then we can import those values that were exported by the other stack by using the import value function and you see an example on the slide deck so in this case we have to combine import value with the substitution function and then it gets a little bit ugly because we cannot use the short form twice so we cannot write um exclamation mark input value exclamation marks up that's not going to work in cloud formation so we have to use the long form for import value and then we apply the sub function and we kind of get access to the vpc id and this is the approach that we use in our aws cloud formation templates repository that is around for quite some time and used by many people it works well it's proven the only problem with this approach is that an exported value is not allowed to change so in the vpc stack if you make an update to that stack that will change the vpc id which is probably a super bad idea anyways because it will recreate the whole environment but that's not possible an exported value is not allowed to change as soon as imported by a stack so that's the only downside pitfall of this solution but still we see this working well in our templates but keep in mind to only export stuff that you don't expect to change so that's our the two approaches that we have and this is actually one of the approaches that i really like um it gets a little bit complicated in really big environments so you have to be careful make sure that everyone kind of follows the same approach everyone prefixes their stuff with the stack name so otherwise it gets a little bit messy but with this limitation in mind it works very well and it's proven and it's available for quite some time so other approaches for reusing cloud formation templates there's a feature in cloud formation called the nested stack so the resource type is aws cloudformation stack you can basically create a stack within a stack and this can be nested so there is lots of room for kind of reusability here because each nested stack can be a template that you refer to that is already existing so for example you can have a database stack you can have a vpc stack you could have no loading stack all kinds of things and one of the examples is our cloud formation modules repository so this contains lots of cloud formation templates that you could easily integrate with each other using nested stacks and the benefit of nested stacks is that you can again pass values between nested stacks by referring to the outputs of a stack you can also use import value export functionality as described before so it really works well on top of the mechanism that i explained you already so we have other tools available to kind of solve this problem um one is the aws cd cdk the cloud development kit and there's also tools like dropper sphere and a couple of other tools basically those tools are preprocessors for cloud formation so they run some logic and they output cloud formation templates and therefore you can for example achieve things like modularization because the preprocessor itself takes care of the modules and it just kind of mixes everything together in the big template and then cloud formation will deploy that template so cloud formation is not involved at all in this modularization um topic here it's it's taken care of by a tool so that's also an approach that works well in many situations but what is actually now this new feature about so what is cloud formation modules it was kind of more or less silently dropped in november 2020 there's one blog post by aws that is kind of weird it misses a lot of pieces it referenced cloud information resources that were not available at that time so when i first read that blog post i was very confused i had no idea how to use that they talked about lots of things that were not explained anywhere and not in the documentation but now the documentation evolved so we have more information and since thomas asked me to kind of explain this feature to all and the subscribers i looked again into the project um in essence a module is a cloud formation template and they sometimes call it fragment in the documentation but it is nothing else than a cloudformation template could be json or yaml both versions or both formats work there is a little helper tool called the cloud formation cli which helps you to create such a module or basically to publish it into and this is the next feature the cloud formation registry so a module has to be published to the registry and so that's a important thing to understand so basically you upload kind of your file into the registry with some metadata a module has a resource type and it is structured like this so you have organizational part you have a service name part then you have a name of the module part and then it always ends with colon colon module all uppercase so that's the convention don't ask me why they're choosing the all uppercase thing so cloudformation is completely pascal cased and so there is nothing uppercase usually in cloud formation but module is so just keep that in mind when you create the the type of your module um last piece here is that you can nest modules but only up to three levels deep so that's a limitation keep that in mind and so with with cloud formation stack as a nested stacks you could have like like the the the deepness of your tree could be much larger here it is limited to three levels so um there's something to keep in mind so don't create modules that are too fine granular so that you have to nest them more than three levels to kind of create the actual application all right so that's it that's the theory uh time for demo i guess that's always the easiest like approach to show how something works so i'm going to switch into my editor and then we will look at actual code here we are this is the first template or fragment um the structure of such a project is always um it contains a fragments folder and within that fragments folder you have a single file and the name doesn't matter and this file contains the cloud formation template and this is a very standard cloud formation template so it has a couple of parameters so those are basically all the inputs into our module at the end and you can use conditions as before and we have the resources block and here i just create a subnet resource for example then i create the route table i connect the route table with the subnet um i create a route i create a network access control list a couple of entries and then and last but not least i um connect the access controllers to the subnet so this is done in the subnet template the cool thing about this is here that if i later create a vpc in multiple availability zones i can use the same module so i will reuse that module to create all the subnets in the vpc that's pretty cool and this is where reusability is actually beneficial where it helps you to have ensure that actually each subnet is exactly the same except for the side range and and the availability zone that it references um to but all the resources and all the configuration will be exactly the same because you will reuse this template or fragment in all of the subnets so the question is how does the vpc module look like and in the vpc module we again have a plain cloud formation template uh so we have only one parameter here so i allow you to specify the class b of the cider range of the vpc but this is just what i exposed to the outside world i mean you could do whatever you wish here um i create a vpc resource so this is where this is done it doesn't really matter what it and actually is about the important piece here is that how i create a subnet because now i use this special type and this is a module reference so this creates a resource based on the module clouded out vpc subnet and that's the module that i showed you before and i do this again for the private subnet in a i do it again for the public subnet in b and again for the public subnet uh sorry the private subnet in b so i always use my module and the module is only kind of created once but i use it four times here in my module so this is already and like demo of nested modules because the vpc itself is a module and it contains subnet modules so that's pretty cool um and it all also works as expected so the question is now okay how can we get this into the registry and this is where the cloud formation cli comes into um play so first what i want to show you is how can you create such a module so let's assume we will create a vpc and with three availability zones so the one that i showed you only runs in a and b so now i want to create one that runs in three zones so what i do is i run the cfn init command so cfn is the cloudformation command line tool and now this starts a little wizard that asks me a couple of questions so first do i want to develop a resource or module i want to develop a module so we talked slightly about the resource approach in a blog post before so when we talked about how you can extend cloud formation so that's where resources come into play so with modules we can only use existing resource types that are already there and with resource you can create your new resource type so that's the difference here but we are going to focus on the modules so i will select m for module and now we have to enter and i like that the type of um our um our module the services will be vpc and i will call this i think i call the other one too soon or two zones maybe i call it three so it's sublet and it has to end with module so that's just oops and this is also one of the things that annoys me a little bit here you cannot really use that properly okay three zones so that's it um the cloudformation cli creates the structure of the project and so you could now start developing it and if you look here into our my editor you will see there's this fragments folder where you can see this json dummy file and i just deleted that and replace it with something else there's also a config file this basically contains the type and there's log file and stuff like this that's not important um so see if an init is for creating a new module so how can we actually upload a new module so let me switch into the subnet folder so this is where my um my submit um my subnet module lifts and i first run the cfn validate command and this will do all kinds of things so first it will extract the metadata out of our module so basically it runs over the parameter section and kind of copies that into another json file i got a lot of warnings here i don't know why i open the issue in github because this makes no sense it tells me all kinds of template validation errors and if i run cfn and let me switch into the folder if i run cfn lint natively um against the exact same template i get zero warnings and what this cloud formation cli tool does it basically uses also the cloud formation linter but for some reasons i don't know it messes things up i see lots of warnings but it's not a big a big problem so it's it still works okay so next step is to submit this module into the registry so the command deceive and submit this will take a couple of seconds depends on how how fast cloud formation is with the registration so let's quickly wait for that to complete all right the module is finally registered with cloudformation if you run this command for the first time it takes a little bit longer because there's actually a cloudformation stack created that contains an s3 bucket kms key in a couple of things to actually upload this into your account and from there it is somehow transferred to to cloud formation so for the first time that will be a little bit slower i will um do the exact same steps in the vpc vpc module so i have to run cfn validate and then i run the submit command so let's do that and again this will take um a little bit of time um until uh cloudformation is happy and registered this um typefast in the meantime we are going to look into the uh cloudformation console and see how all of that looks like there so this is the cloud information console and on the left you will find the cloud formation registry where you can click on modules and here you now can see that i uploaded two modules the subnet module and the two zones module so that's exactly kind of reflects what we did and before so what i'm going to do now is i'm going to deploy my demo template and i will show you how this looks like before so this is my demo template it's super simple it only creates one resource at least that's how it looks like and it references the vpc2 zone module that i just uploaded and registered and it sets the class b to 42. so i mean you can enter whatever you wish here until unless it's it's a number between 0 and 255 so that's what we specified in the parameters section um so let me do that let me create a stack and i will do this using the ui here quickly so we'll upload the template file and i will select the demo template and now the usual cloud formation steps are needed so we click through all of that we acknowledge all those things and then it is started it will provision our resources and now it gets interesting because if you look at the actual resources that are provisioned you might be surprised what happens so it takes us some time here confirmation just figures out all those modules and the funny thing is that what they do is they kind of extract all those modules and expand them into the main template so that's all what they do and they do this in a very crazy way um so still work in progress so what's going on here it's okay now it gets started um we are starting with the vpc and as you can see the logical id looks very confusing and what they do is they actually prefix like when they merge all those resources from the modules into the like top level template they prefix the logical id with the logical id of the module so in this case the modulo logical id is vpc and the logical id of the resource is vpc as well that's why it's now called vpc vpc and this is very error prone i mean if you create a resource in your main template that has the logical idv pc vpc then we get into issues because then there's kind of a collision in logical ids so i'm not very happy with that approach and so not sure why they merged the logical ids in that way um and um but that's what cloud formation does and this is also how you would reference those resources so basically in your main template you can reference something that doesn't exist in the main template you can basically reference into the modules by um creating a logical id based on the module modules logical id append the logical id of the resource and that's it and if you have nested modules then this goes on like like the chain goes on and this is completely mad in my opinion but that's how it is so as you can see it's working it creates all kinds of resources here as subnets and stuff like this and this is great so that's what we actually expected from this tool but i haven't expected when i used it the first time is the way it it deals with logical ids by the way the same happens with outputs so the outputs are also merged together in this kind of crazy way um but okay that's something we can live with and just keep that in mind when you create your logical ids to not kind of interfere with the namespace of the modules all right so this thing is now completed um all the resources are there so we can now use this vpc great uh so modules are working so that's actually the good news but now let's look into the bad news so as i promised in the beginning i will talk about what cloud formation modules is about the good parts and also the bad parts so the good part actually is it works as expected and but now we will talk about the bad parts and this is actually a very long list so i'm very sorry for that but it really has so many limitations that i really find it hard to make anything useful out of it so first parameters are not validated so in a cloud formation parameter you can specify for example that this has to be a string with min length and max length this number has to be between min and max you can specify regular expressions you can say these are the allowed values like all kinds of of validation um or per or like configuration can be made a module ignores all those information completely so if you pass in something that's not valid it will still work i mean that's kind of not so cool um import value is not going to work inside a module so you cannot use the import value function in a module you can also not export an output in your module it's not possible if you want to reference a resort inside a module so the logic that i just demonstrated you applies so you ref the module id and expand the logical id of the resource same works for if you want to get an attribute so this is i mean i don't think that this is a good approach because there is the potential of collisions within the logical ids because you can create the same logical id here and then it will kind of blow up next part um you cannot create a cloud formation stack inside a module so basically nested stacks are not supported in modules also snack sets are not supported that might not be a big problem keep in mind that you can only nest up to three levels deep with modules and and what actually is more or less the bummer here is that you have to register the module in each account and each region so imagine you're working in an enterprise and that's one of the use cases that aws kind of uh has in mind when they talk to us about modules yeah that's a good tool for enterprises where the like the cloud the cloud excellence and how center of excellence can kind of distribute this is how we want to create a vpc to the whole organization and i i really want to see how this works because in each account in each region you have to then deploy this module and if you want to create a new version of the module you do the same thing again so i don't know um this is kind of a weird approach as far as i understand it is possible to do it nowadays since a couple of days with stack sets and because in cloud formation you can now create the resource the registration resource to register the module in the registry but i mean it would be super cool if this would just integrate into the organization so you could just take it share this module with my work or share this module publicly that would be even cooler for example for people like us who want to distribute this to all the people but that's not possible it's super private within account regions and so keep that in mind and the management overhead is is is not um or is quite significant all right so what is not working as well transform or macros are not supported and the most i think popular one is the serverless application model trans um transform so it's not working inside a module and cannot use it the same applies to the transform function which includes the aws include so you cannot include other files inside such a module not possible and other another pitfall where i was running into when i first i was trying to create kind of a serverless example here it and it kind of stopped working when i use the adore cloudformation package command which usually turns relative paths into s3 urls and uploads the file and this does not go into a module so aws confirmation package doesn't know about the content of the module because this whole merging together is only done as soon as you create a stack so the tooling doesn't know about it and that's why they don't upload or turn relative paths inside a module into an s3 url so it's not working um and i think this kind of eliminates all the the serverless use cases so you cannot do anything useful um with modules in the serverless space so um that's also kind of bad news so yeah that's um kind of the summary here um i'm not using cloud formation modules and i'm not going to use cloud formation modules in the next um or in the near term because of all those all those limitations that i just talked to you and if you are interested in a video about our cloud formation templates or the cloud formation modules that we provide then i'm happy to talk about that and give you some demos and show you a little bit how it works so if this could be an interesting topic for you let us know and you can write a comment uh in the community you can send us an email and reply to the newsletter and do whatever is is is the easiest way for you to get in touch with us i would really like love to talk about all of those topics so i'm just looking for your input in what's most uh relevant all right so let's um close this video um so i think i explained to you everything that is important about um cloud formation modules so if you have any questions um around this topic or other topics and community.cloud and out io is exactly the place where you could ask those questions um so it's a private there's a private discussion forum for cloud or not blast subscribers so if you see the list of forms click on the cloud or not plus forum and you can ask your questions and only people within the community within the private cloud or not plus community will see it and we will answer your questions um or other people's from the community will join in and and provide their advice as well so thank you very much for watching and don't forget to like this video if you learned something new your feedback helps us to produce relevant videos you can reach us via twitter email you will find all the details in the video description below so we are back in one week thanks for your support bye

Channel: cloudonaut

Views: 333

Keywords: aws, amazon web service, cloudonaut, cloud, cloudcomputing, cloud computing, aws training, aws cloud, aws tutorial, aws tutorial for beginners, amazon aws tutorial, aws cloudformation template, aws cloud formation template, aws cloudformation modules, aws cloud formation modules, cloud formation, cloudformation, modules, cloudformation modules, cloudformation modules explained, reuse templates in aws, reusing templates in aws, aws templates reusing cloudformation

Id: PM_f-LWdQgc

Length: 28min 24sec (1704 seconds)

Published: Sun Oct 24 2021