Custom cloud infrastructure as code with AWS CDK - CloudFormation Custom Resources Lambda Backed

hello world and welcome to another episode of uber sometimes we have more infrastructure resources than the one that aws cloud formation provides it's quite common that we have a lot of resources that we consider part of our infrastructure that we need for our application or our architecture to work and confirmation doesn't have a resource definition for it so what we tend to do well some cases we write a script um in a paper or in an excel file and we put okay now we need to go to the console and i don't know create a database and put this information in it with these properties or we need to run this bash script or we need to even create a setup lambda function that's quite a use case i have seen lately that a lot of people have land functions around the setup configuration so whenever you deploy a stack the first thing you do is to invoke this setup lambda function those are really great solutions and they do their job but if we go to the idea of infrastructure as code they kind of belong there but it kind of ruins the purpose of having this script that you can basically throw into any aws account or in any region and it will replicate the like environment and everything you need so in today's video i want to talk about aws cloudformation custom resources so aws cloudformation custom resources will solve all those problems that i mentioned everything can be represented as a cloud formation resource even though there is no definition for that particular resource provided by aws and you can do that by using cloudformation resources in this way you can build your whole cloudformation template with all these custom resources that are like i don't know creating your database and configuring it and i don't know using uh location map uh location services that doesn't have confirmation support or you're going to see in the demo uh recognition collection for faces and all other things that might not be aws infrastructure whatsoever create a new sandbox account in a third-party api you're using or doing something on-premise even it doesn't matter but all of these can be attached as cloud formation resources so when you put the stack up you deploy this cloudformation resource all these other things are also uh getting installed or getting run or whatever these custom resources are doing and when you tier they stack down you remove this confirmation stack all those resources revert back to nothing so custom resources basically enables you to write custom provisioning template or logic in the cloudformation templates that will run anytime a stock creates update or deletes so these typical operations that we are going to do whenever we create a new stock we run a i don't know cloudformation uh deploy the stack then the custom resource will deploy then when we do an update it if affects the custom resource we can update it or if we do a delete we want to delete that custom resource and it happens as well there are many kinds of custom resources so i want to focus on the lambda backed ones and i think these are very very powerful so these are i think something i've been using now lately and they're great so basically you will associate a lambda function to a custom resource so whenever the cloud formation is deploying for example it will trigger the lambda function automatically so this is very close to this setup lambda function that we have that i mentioned at the beginning that is quite a typical use case that you have that lambda function to create all the basic things that you need in your application but cloudformation doesn't provide so now this is integrated into the deployment of the stack and you don't need to remember to invoke that function it will happen automatically it will be invoked by cloudformation and also it will be invoked the other way around when you want to delete it and that's very handy because it happens quite often that you create a lot of custom resources with this setup uh scenario that you're launching invoking the lambda outside of cloudformation and then you first need to remove all these things that you created and then you can remove your stock or you remove your stock and you forget of all these other things that keep on hanging there laminar function is extremely powerful because it allows you to interact very well very fast with other aws resources you can use the aws sdk there super natively so you can invoke basically anything but you also can use third-party services or on-premise services by using http connections so that's what it make it so powerful in the demo today i want to show you how to define cloudformation custom resources using aws cdk i know it's very specific but i've been going into cdk lately and i'm really enjoying it and cdk has a custom resource provider construct that is like a mini little framework that will be taking care of a lot of the things that custom providers need in order if you define them raw cloud formation so basically it takes care of a lot of the complexity of of deploying and managing these custom providers so before going to code i will explain you what i want to do because it's easier so i want to build a custom resource for creating a new face collection using recognition so when you're using recognition to identify faces you can store those faces in these recognitions collection of faces and then if you pop a face it can search around i'm making an interesting demo for you but this is another video in the future but the idea is um you create this collection so when i started uh building this uh application in cdk i was like well i can create this setup lambda but why not to use a custom resource because this create collection is not an aws cloudformation resource it's not available as a resource so instead of that i will be creating a lambda function that is using the aws sdk to create the collection and also i will be using the aerolia sdk to remove the collection so when i do cdk destroy the collection is gone and i don't have random collections jumping around because we don't want that so now we can go through the code so as always i will start with an empty project and i will be sharing the um the code with you in github so before jumping into the code if you are here watching still watching and you are about to use my code from github and all that stuff like this video share this video comment this video we want people to get to know this channel and that's the trick for making youtube recommend my content so now let's go to the code so as always i will be starting with an empty directory that we are going to create a new cdk application you need just to start from a cream slate you can do this in any existing uh cdk application so yeah i will be using typescript you can do this with whatever you like as always but i'm a javascript developer typescript so it's easier for me so let's wait for the application to get uh kind of downloaded and created i will be then going into visual studio code and showing you what to do there so let's go to the code and start uh defining what we need so we're going to our um leave folder and there in that stack we are going to create our custom resource we don't want to put it anywhere special this is an empty project so well if you're building a custom resource inside your stack just figure out where you want to put it so the first thing we are going to do is to install all the libraries we need i'm pretty sure i will forget some library and we will need to install it later but we need to install lambda because that's something we are going to use we need to install the custom resources because that's a library we are going to use we need to install the logs because we are going to be talking about logs at some point we are going to find some lock retention policy so that's something we need to install and later on we will need i am but i think i forget to put it in this time and we will do it in a second i will install it in a second so when all the libraries are installed now we can uh move to start writing the code so the first thing i want to do is to define a constant that is the name of the collection um this is the recognition collection i just put any name because this is for the demo uh you can use basically any resource here it's built i will just pick a recognition collection face collection because i was building this hour demo and i like to utilize stuff then we are going to create a new london function and this is going to call on event you can put any name but i like to make it a line with the uh where this lambda function is going to get called in the custom resource so there we are going to put a name for it recognition custom resource function something small and simple and then we are using a runtime uh no 14 because it's available now and we can use it we can also set the handler where the code is and um all the basic things from alarm function so we have not yet built a handler but there we will put the code that we need to create the collection using aws sdk you will see that it's very simple and that's not the point of this video so i will be storing all my functions in this functions directory and i'm making the reference there and i'm need to import the path library in order for this to work then we need to pass an environmental variable that is this collection id that we just created because we will need that id later in our code to create a collection with the name that we want and then we can pass a row lambda functions are born without any permissions no matter if they are custom resources so we need to give permissions to the lambda function to be able to create our collection in recognition so we are just creating a new role and i'm basically need to import i am here in order for this to work and after i import i am we can go back and uh do some definition of roles so this is uh the role for the lambda function and i will be giving some policy to the role so we will add to the role the capability of create a recognition collection and then uh to anything in recognition because we don't have any resources yet so so then i attached that role to my loved function and then we can move on to start creating the custom resources the first thing we are going to create is a provider and this provider is this kind of mini framework that will be handling a lot of the managing of the custom of the custom resource so here we will need to attach the lambda function that will get triggered on an event an event means when the stack is deployed updated or deleted so now i will be doing the same operation whenever and you will see that it will fail on delete and i will need to make some magic but that comes later on the video also if you have a lambda functions that are asynchronous here you will have to do a callback function so you can also attach that in our case we don't have an asynchronous call but in some cases your operations your infrastructure might take more than 15 minutes to execute and you might do it asynchronous you may start the lambda function and then leave it running or invoke something overall and then you might need to have a callback operation and you will find that here if you need so and the provider will take care of that also we are going to find the lock retention i will put it for one day whatever you need uh for this provider so we get some logs if needed and now we can basically add that provider to as a custom resource and we can have everything we need the cool thing now is that if we want we could have multiple instances of this provider that will be launching this lambda four times i don't know why but who knows what your use case is so they're all independent and this is quite great but yeah so now let's create the custom resource and pass the uh a name for it again mission custom resource and here if you have multiple just put different names and then we need to put the service token and this will be basically the air and end of this provider but there is an api for that so with all that then we have our custom resource created and the mini framework that will manage the creation of that using cdk so this is pretty neat uh we need the code for the login function before deploying i always forget about this so let's create that and then we can see how this works in action and don't go because i want to show you also there how to delete the stack because i think that's important when you are building infrastructure as code you want it to be easy to add and also be easy to destroy so uh we will create the lambda function in this case we are creating a recognition uh collection using the italian sdk and basically we're just calling that and that will create the collection for us no big deal so let's go on and deploy these into our aws account so for doing that first we do npm run uh build to compile everything and then we will do cdk deploy and this will deploy the whole stack that only has one custom resource well you will see that it has more things but in our head it only has one custom resource and this will deploy everything into our aws account and let's fast forward in order for seeing what is the result so now everything is deployed let's go to our aws account into management console and see what we can see there in the cloud formation stack we open the cloud formation service and there we will see the last stack if you just deploy that as last is the name of the thing that you have deployed if you go to the resources you can see that there is a couple of land functions and some custom resources and some permissions and things like that so we can go to the lambda function um console to see what is going on there and i think it will be a little bit easier to understand so let's open lambda and applications and we go to the applications with just deploy and there you will see that there is two lambda functions and you're like why there are two on the functions i just deployed one lambda function that is the custom resource what is this other line function well that's the provider that mini framework that i was telling you that is in charge of uh handling all the lifecycle of your custom resource that cdk created for you so if we go in there we can uh we can see that there is a lot of code that we have not written so this is cdk doing its thing so that's what we love it so we are not getting into there uh and then if we go to the other function you will see that the code you wrote well that i wrote that you just copy paste or if you change it it will be the code you wrote the one creating the collection in our recognition good then we can use the aws shell to check if our collection was created i will be using the aws cli for that in the lds cloud shell and i will be running the list collection command to see if my collection if this collection that i just created appears in the recognition collections that i have so we can see that the demo custom resource is there and basically our collection is listed so what happens now with this code if we do cdk destroy it will explode yes it will explode because our uh custom resource doesn't have anything uh basically is trying to create another collection with the same name basically on destroy because it doesn't have any way to know which event is on create or on delete so i will remove everything magically i will remove the collection recognition phase collection using aws cli i will be removing the stack and we will start all over again well not over again i will show you what you need to do in order to make the delete the destroyed work so the first thing you need to do in order to get the destroy to work is uh in your stock infrastructure add another uh policy another action to your policy that is the recognition delete collection because we want that on the delete of the cloudformation stack the collection gets deleted as well so good we have that we add that and then what we want to do is in the code of our london function to understand what is the event that is coming in and that will be coming in the event object itself so we know that lambda functions the event object of the lambda functions depends on who invokes the lambda function in this case is this a custom resource provider that is invoking the lambda function and on the event it will be basically sending a request type and if the request type is of the type um oncreate well in create it will be creating the collection we want that collection to be created good but if the request type is delete we want the collection to be deleted and now we have that logic we are building that logic so whenever we do cdk destroy then the request type will be delete the collection will get delete and there will not be an error on the destroy part so let's fast forward and deploy this thing and see what happens [Music] so we are deploying this thing again and when it's deployed we can go and check so again i list all my collections as soon as this finished deploying everything appears there and now i will do cdk destroy previously it exploded it failed dramatically but now we will not have a fail it will just execute lovely and when it is finished we can go to our cloud formation stack and see that that confirmation as stack doesn't exist so that's good we can go and list all our collections and you will see that that collection that we created doesn't exist either and this is what you want from uh infrastructures code you want it to be easy to tear up to create and also to tear down so i love custom resources i use them a lot and i think now with this cdk integration is so so easy to do them the provider is so powerful it does so much work for you so i hope this video inspires you to try this uh thing out so that's all i wanted to share with you today if you have not liked this video you just watch half an hour of content and didn't like my video what are you waiting for go and like it and let me know in the comments box below what other cdk content you would like to see i know you are so into cdk content lately that i would love to build more about this so let me know in the comment box below and i see you in the next episode of football you

Channel: FooBar Serverless

Views: 1,700

Keywords: foobar, infrastructure as code, cdk, aws, amazon web services, aws cdk, aws cloud development kit, aws cloudformation, aws tutorial, aws cdk tutorial, aws cdk demo, amazon cdk, devops, aws cdk typescript, aws cdk typescript tutorial, aws tutorial for beginners, aws cdk tutorial javascript, aws cdk tutorial typescript, aws cdk typescript example, cloudformation custom resources, lambda backed custom resources, cdk custom resource lambda example, cdk custom resource

Id: u7FdDFta2XI

Length: 21min 57sec (1317 seconds)

Published: Thu May 27 2021