Building a Heroku-like experience with Pulumi + AWS in 60 minutes! ⏱ [DevOps Office Hours Ep. 10]

hmm so hello and welcome to the 10th ever episode of the devops directive office hours uh it's been a little while but i'm i'm glad to be back and i think we've got a fun episode today i will be working with palumi and specifically the automation api within pollumi and so for today's guest i have lee briggs who is a developer advocate with palumi how are you i'm good thanks yeah just back from vacation so good to dive into something fun uh before uh before yet another vacation um i've i've been at palumi now uh over 18 months which uh you know in the current world climate feels like it's been five minutes um but um we're going to be looking a little bit at um the different ways that you can use infrastructure automation to provision uh the infrastructure that your application is dependent on um and i'm really excited about doing it we've uh we're gonna be using python today uh poloomi supports multiple different language sdks uh i think we had a chat beforehand and you know python is something that you're pretty familiar with uh so hopefully some of the concepts that we'll be kind of going through will make sense uh and i think it's gonna be fun experiment awesome yeah i think most of the audience is probably familiar with the concept of infrastructure as code but now adding this element of building an api around that to abstract a little bit of that away from the development team is an interesting one that i think will be a cool thing to dive into yeah and i feel really passionate about this like as um as a little bit more about my background i started out as in the industry as a kind of system administrator so writing python and bash scripts uh 10 to 15 years ago um and obviously as the industry has changed kind of moved into cloud computing and containers and all those other exciting things that you cover on your channel every week um and the the kind of traditional way that you would provision infrastructure is that you would write a bunch of configuration and then put it into a ci cd pipeline whether it's github actions or amazon code pipelines or something like that and i think this approach kind of is a little bit more user-friendly especially if you're supporting those users who are more used to writing applications and they still need to provision infrastructure what you can do is is abstract some of that complexity away to make their lives a little bit easier um and hopefully you'll see an example of what that looks like the final thing that i'll say before we dive in is that um you know if you're not really familiar with if you're familiar with kind of doing devops-y things with infrastructure uh using configuration languages some of this stuff might seem a little bit intimidating at first but i'd like to just say that i felt it was intimidating once upon a time and i felt that like kind of like writing this infrastructure stuff in in these languages really was a little scary but you know kind of getting hands-on doing something that i already knew ie cloud computing in a language that i wasn't necessarily familiar with like python uh you know i've really leveled up my skills and i feel a much better and i'm now writing infrastructure configuration in four different languages on a day-to-day basis so you know if if you're one of those people who is is following devops directive to learn every day i think this is a great opportunity perfect well i'll pull up our screen share and we can dive right in anyone who is watching along feel free to ask questions along the way and i'll pull them up i see carlos in the audience hey carlos glad to have you here um yes i think i'm i think we decided i'm gonna drive today um so there's a couple things that i've done to kind of make sure before i actually get started there's a couple of things that i've kind of taken care of uh beforehand um and so the first thing to be aware of is that i've got some valid aws credentials so i'm logged into aws and i can actually uh provision things and i can do aws s3 ls um you know i can list a bunch of different book buckets and you can see so my colleagues are making use of the cheap s3 buckets in aws um and then the other thing that i've done is i've decided where i'm going to store my paloomy state and there's a couple of different things that are available to you here the command that you use is paloomi login and by default we um we will store state for you we have this free tier um that will allow you to stare store resources it's free for individuals um and it's free uh up to a certain number of resources for teams uh if you do not want to do that and you want to stick down the open source path you can either store your state locally using the login command or you can use um an object storage like aws s3 google play cloud and azure um to actually store your state i'm going to use the paloomi servers today because it's just easier and i don't have to configure an s3 bucket to do it um and so i can verify who i am by typing the polluli who are my commands so i'm logged in as my user um obviously i've installed the pollumi cli as well like that's important uh but it's actually going to be interesting because we're only going to use that for the first part um make sense it does one quick question comes to mind uh presumably if you're using the hosted version of the state management it handles locking for you uh is that true or does it not well it does support locking for you but also um as of um about three or four months ago the locking is also handled by the object storage as well you have to set an environment variable which i do not know off the top of my head because it's currently in preview um but that's what that will soon be part of the the cli as well um the reason that we didn't support uh locking in the open source storage before is because not all um s3 uh compatible storage was uh strongly consistent right s3 with aws you have to do a dynamo for terraform at least you have to do a dynamodb instance yeah a third thing but like uh amazon did a huge blog post which is like a huge technical achievement where they made s3 strongly consistent and so now we use s3 as the locking mechanism as well and it works great lots of our users are using it um and so you know you get all the benefits of locking making sure that the only one person is updating a stack at a time basically super cool we also have lots of other awesome features in the service i'm not going to talk about those today but i would strongly recommend trying it out you know i got to put my polumi employee hat on for just a second there um okay um so what we're going to do is we're going to bootstrap our paloomy program and this is generally the the path that you would take in most situations so you'll see i've created a directory here called devops directory of director devops directory a directive it's empty um and so if you think about how you would set up a normal python program you might create a requirements.txt you might create a virtual m all those things are general things you would do in a in a python program uh palumi is going to do a lot of this for us um so what i'm going to do is type polumi new um and if i type puluminu you get this lovely drop down which will show you all the different languages and templates that we have i am going to choose aws python if you already know what you're going to use you can just add this after the paloomy new command so you can just type paloomi new aws python um and if i hit em to here it's gonna start asking me for some um for some uh information so uh it's gonna ask for a project name by default it sets it to the name of the directory um i'm quite happy with that so i'm gonna hit enter um and then it actually for a description super useful if you're doing uh collaboration things like i've certainly run into infrastructure's code implementations where i'm like what is this actually doing um so a description is usually pretty useful because we're just gonna uh do an example today i'm gonna use the default and then the final thing that it's going to ask you for is a stack name and stacks are reusable implementations of pollumi um we're going to try that out in a little bit if we have time uh just for the for the sake of this particular um exercise i'm just going to use the default of dev and what's going to happen is it's going to then prompt me for the required inputs so like the polumi sdk sorry the aws sdk has some required uh information and palomi will like say we know that we need to um set the region i'm going to use u.s west 2 because i have had some bad experiences with u.s east 1 in the past and so i'm going to use us west 2 and then what's going to happen is palomi's going to set all it's going to set all the boilerplate up for us so it's going to create a virtual environment install all the dependencies in that environment for us and create all the files that we actually need to actually get started which i think is a pretty nice experience uh i am biased of course uh but it's much easier than kind of having to remember a bunch of different commands that i have to run in order to get started um so this paloomy new command is usually the first step in kind of getting off the ground with any paloomi project cool yeah it makes sense and then we can take a look at what it actually installs in the requirements file right yes uh so i'm just gonna wait for this to uh finish uh pulling from the internet uh obviously it's gonna pull a bunch a bunch of different uh dependencies some of them are already cached and then some of them aren't um and it's told me that my project is ready to go um so i'm gonna switch to my vs code ide and uh you will be able to take a look at some of the different things that is adding in here so in our requirements.txt we have uh two dependencies uh just the paloomi sdk and then the pollumi aws sdk which allows us to actually interact with aws itself we have lots and lots of different providers so if you're more familiar with azure or kubernetes or google cloud or digitalocean or anything like that all of those are available and it's just as simple as adding um you know polumi dash azure in here and uh setting your constraints and uh and and doing a pip3 install and you will get all the dependencies installed for you which i think is pretty awesome nice um the other things that um are kind of unique to paloomi and not necessarily a python project you get this pollumi.yaml which will have the project name and if you remember our description as well the one thing that we didn't specify is the runtime this is uh because we support multiple different languages um you know we we have to actually specify to plummy which runtime is is going to be used um and then there's some options so if i have if you are the kind of person who has a global virtual m the um you know you want to use for everything you can specify which one you want to use here um i personally like to keep these constrained to different projects so that i can manage my dependencies a little bit more carefully and then uh there's another yaml file which is our stack configuration and so is that just a relative path to where that vn is living okay yeah so we can see here we've got this vm directory here right so um if i wanted to do home vm or something along those lines then i could do that uh but right now it's just in this vm directory um and then um the final thing is that we we set our aws region and you'll notice this went into us into a file called pollute dev.yamo um this this configuration is stack specific um and so what that means is that you can create a polumi program and you could do something like targeted at a dev account and then create a new stack and target a production account and you can reuse the same code and so uh depending on how much time we have left at the end i'll be able to show you an example of that perfect okay so uh we've got a very very simple uh polumi program uh that it's created for us and all this is gonna do is create a my bucket program and export the bucket name we can have a look what that looks like uh clear pollutant up presumably that's going to fail because of the name of that bucket or no it's going to generate a random name and so if we take a look at the details here uh palumi does uh something which is interesting in infrastructures code as it adds this uh unique prefix to every resource unless you explicitly set the name um and the reason it does that is because lots of different cloud providers have even more immutable apis and so what this means is that by adding this random prefix at the end um we can say that if we ever make a change to that particular resource that is immutable it can create a replacement for perhaps behind a load balancer um and then um kind of take down the other one afterwards which will kind of allow you to do lots of blue green deployment um instances um so this is the obviously the most basic of palomi program it's just an s3 bucket um we're going to be a little bit more uh elaborate today in our pollumi program because we want to make something that's going to be reusable um a bunch of different implementations so the first thing that i'm going to do is use polumi's um reusable uh component mechanism so um let's take a look at what that would look like in action so i'm going to create a new file nope that's a folder lead come on uh we're going to create a new file i'm going to call it webapp.py it's just a standard python file and i'm going to import my dependencies into here so i'm going to do import pollumi and then import pollumi aws um and then what i need to do in order to make this reusable is i start need to start needing to use some object orientated programming concepts and if you are a if you are a more of on the system administrator or devops engineer side of things when i say that i'm going to start using object orientated concepts you might immediately panic and think well this is a little bit too far i um i come from that background too um and i'll be honest i did a computer science degree and somehow managed to get through it without understanding what object orientation even was um i'd you know you see lots of these examples like a dog or a person and it's very rarely kind of applicable to a real world scenario and it's often difficult to actually understand what the actual concepts are um and so what i'm going to do is show you like a real world example of a reusable component using this object orientation using pollumi components which hopefully will make a little bit more sense for your cloud kind of like mentality and so let's see how that goes a pollumi component has uh two um parts of it it has the the input properties the things that you want to configure and then it has the actual resources that you're going to create um and so i'm going to create two classes one called web app args like so and and i just saw donnie in the chat says i mean most developers don't even know what object orientation is so that's perfect we're in good company absolutely um and then because this is a object orientated um mechanism uh we need to create within python we need to create this init definition or this init function here and obviously um vs code is a little bit ahead of me in terms of what it wants to do um and then the final thing that we need to add to this is the input property so for our reusable web application the only thing that we want to allow people to configure is the actual image name because we're going to be really opinionated in how this works and looks we're not going to allow them to do you know if you think about how you run applications in amazon web services or something like that there's so many input properties and it can be so overwhelming and because where the we're the devops experts because we're obviously followers of devops directive um we are going to be the uh the best practice implementation people and we're going to say you're going to have a docker image and we're going to decide on how to do the rest of it we're going to kind of do this opinionated implementation so all i'm going to allow people to configure is the is the image like that's the only thing that i'm going to allow them um and so what we need to do is just make sure this is actually used so self.image equals the image from the argument and that is our first instantiation of anything in terms of a pollumi component so this is just the actual arguments that are going to be passed to a uh polumi program or a polymer reusable component what are how you how are you following along so far sid how is it looking looking good looking good we've got one question we've got a couple questions in the audience but i'll bring up the first and hagie says so you're going to create an instance of what you initialize and yeah so the whole idea with object-oriented programming is you have this class definition and then we can create many instances of it and each one can have a different image name so when we instantiate it we'll specify that image and then it'll be tied to that instance of the class and then another question more broadly that i'll throw over to you is what's the difference between palumi and terraform my favorite question um so if you've used terraform or other infrastructures code tools there's other things out there like azure resource manager or um aws cloud formation um those um those infrastructure code tools use configuration languages in order to actually define your infrastructure so you may be familiar with terraform and you've used hcl before hcl actually is a domain specific language which actually compiles down to json and it's kind of native uh components and i think that compiles down to maybe a little bit maybe a little bit of a hand wavy answer um but ultimately uh they use configuration languages that are kind of uh procedural in order to actually define your infrastructure pollumi uses um application programming languages in order to define your infrastructure so right now we're using python we alright we also support javascript languages like javascript and typescript and i think you did a uh um an awesome code ai um implementation with javascript a few weeks ago which i thought was awesome and we also support the go programming language and the dot-net framework so we also support c-sharp f-sharp and my personal favorite vb.net um yes that's still a thing what's interesting about palumi is that you're using these kind of languages that are generally considered imperative so you would do things like if statements and conditionals in order to actually make sure your operation you know kind of proceeded in the way you wanted it to however with palumi we're going to use these imperative languages but it's still going to have a declarative end state so it's still going to actually drive towards a desired goal um even though we're using an imperative language and once we've kind of fleshed out this particular implementation that will become pretty clear um one of some of the benefits of that are that you get a lot more flexibility in the way you define your infrastructure for example you can use things like object orientation if you really want to uh you can do things like unit test your infrastructures code so like if you've used other infrastructures code tools like anything that's yaml based it's really really hard to test them without actually provisioning your infrastructure and checking that it looks the way you want it to with palumi you can write unit tests that will actually pass and fail when you make configuration changes once you get to a certain level of complexity that's super super useful um because you can just unit test the changes that you're making and feel way more comfortable in your infrastructure and then finally you've got conditional logic so like loops and conditionals and if statements and all that kind of stuff are all available to you so hopefully that's a good answer um i feel like i've answered that question a thousand times and i never do a great job of it but hopefully that gets us there no i think that's clear i think the the fact that you can use whatever testing framework you're already familiar with for your language of choice is is a really unique thing uh that using these application programming languages brings to the table and and i think another thing that i have felt as my in my time prior to working at pollution work as an infrastructure engineer is i've implemented lots and lots of different infrastructures code tools whether it be kubernetes yaml or cloud formation and i'm working with this large team of developers that's already writing application code so like python or java or all that kind of stuff and they feel really overwhelmed and intimidated by having to learn this just main specific language that's only reusable for one thing um and i i part of the reason why i actually discovered pollumi is that i was really trying to help enable those developers to be able to actually write their infrastructure code um and it really uh made a difference the language that you're using really does make a difference it's really easy for us as devops engineers to go well you know um json's really easy to understand or yaml is really easy to understand but for a developer who's been writing everything in java their whole lives that's not necessarily the case right true okay so um we've defined our arguments for our web app so now we actually need to define the web app itself and because we're going to want to um you know do something that's relatively straightforward i've decided to use fargate today in order to actually deploy our application so our web app is going to be deployed in fargate um using all the kind of aws mechanisms so what i need to do now is define a web app class which extends polumi.component resource so like we're just going to say a web web app is a type of polumi component resource and similar to what we did with our uh web app args class we're going to define an init definition so like these are the things that you will need when you instantiate this and what we're going to do if we take a look and you'll notice that i get all of the benefits of using kind of intellisense in my ide and so i'm going to define all the things that i actually need so we're going to have to define the type of this resource the name some properties and whether it's remote and so what we're going to do is specify a name which will be a string and then we're going to specify an argument input which of course is our web apps ads because we've already defined this right um and then we are going to define some options which will be pollumi dot resource options and by default we're not going to need to need any of those so i can set a default here like so um and hopefully you're kind of getting the idea of what this looks like you get a lot of help from the ide when you're doing this right like if you've ever written any yaml before um even with all of the best uh input vs code extensions in the world it can really be a little bit of trial and error right like you really don't know necessarily what you're going to be doing the final thing that we need to do here is like a bootstrapping is we actually need to define what we call the token it's really hard to explain what this neces what this actually is until we get to the stage where we're instantiating this so i'm just gonna fill this out real quick um this is pretty much boilerplate if you're ever gonna write one of these components yourself it will make a little bit of sense um but this token is like a unique identifier within pollumi that says um you know that we have actually instantiate one of these things when we run our pollumi program in a few minutes i think it will make a lot of sense okay yeah okay so what we'll do now is there's a couple things that we need to grab from our aws environment um in order to actually um know where to deploy our web application so i need to set up the name first so self equals name which is of course this property here and then we're going to grab a vpc so i'm going to use the default vpc for now because um you know i don't necessarily want to um and i've just realized i made a mistake so my id is helping me out here and saying hey lee you screwed something up you didn't do something correctly it doesn't actually know what aws is so let's fix that like so and is that a standard convention to take your provider and map it to the name of the provider that's what i like to do some people like to use the from paloomi aws import s3 mechanism like that i personally don't like that i like to be a little bit more explicit in my code because i think it makes it more reusable when other people come along that's the only reason so this is again because we've got the flexibility of the programming language um you can kind of make your own determination on what you think is best um i i like to do it this way i think it's easier to read and easier for other people to follow along um okay so we're going to say that we want to grab the default um vpc and again look you'll notice that my uh ide is really helping me here so i'm going to scrap the subnets as well which we need to know aws.ec2.get subnet ids look and i can grab the security groups i can grab all different kinds of things um if i want to um and then it will also tell me what the properties that i need to add are so vpc id equals and i think this is an important time to kind of note something as well um when you create a pollumi program it has it makes api calls out to pollumi that say sorry out to aws it says ask in this particular case it's going to ask um aws what the vp what the default vpc is i could look that up and hard code it and if you think about if you're using something like yaml and like cloud formation you might have some fancy functions in here with pollumi all you need to do is pass the variable output so like if i do dot here you'll see that it's grabbed the vpc and i can just look at all the different outputs that are available here and say that i want the vpc id to come from here and so it's a really kind of easy and straightforward mechanism i certainly think so anyway yeah and so the vpc there is a class that's defined on the ec2 class within the pollumi aws provider that's exactly right yeah um and so i'm just setting it up with inside this component and i could obviously write this directly in my main.yaml similar to what i've done here um but of course like i want to make this reusable so i'm going to kind of go straight from the get-go so if you've ever deployed anything in fargate you'll know that the first thing you need to set up is a cluster so let's define a cluster aws dot ecs dot cluster and i can of course use python f strings in order to actually define the name of this cluster so i'm going to call it name dash cluster like so um and then i'm going to do something that's kind of unique to pollumi is i'm going to make the output look pretty um which is of course always fun so i'm going to set the resource options i'm going to set the parent of this equals true again when we actually run our paloomy program it'll make a little bit more sense um and i don't want true i want to be self so um it will make a little bit more sense why i've done that in a few moments um because i think uh without actually kind of showing you the output it doesn't really make sense why i've set this resource option here but we'll we'll do another resource option in a second okay make sense cool i'm gonna cheat now um because i'm looking at the time um i'm loneliness it's 9 21. um and what i'm gonna do is uh we we now need to create a security group um and what this generally looks like is you do security group equals aws.ec2.securitygroup and define it in the same way as our cluster however this has got a bunch of different inputs and so what i'm going to do is copy and paste it simply because i'm a great software developer and i've found this on our stack overflow so i'm gonna do this um so i'm gonna talk you through what i've actually done now um simply uh to make things a little bit quicker this is mostly just to allow traffic to get to our our code right exactly but i think there's some important things to kind of make a note of here uh we've got the uh security group id which i've used as an f-string um and i've also uh like the security group itself when you create it it needs to know what vpc that it's going to go into right um and so again i've used the vpc id like i did here um which has been kind of retrieved from this get vpc call what i think is interesting about palumi is we already talked about how it's imperative um and how like sorry how it's declarative instead of imperative what this is actually telling our security group is that it shouldn't create this security group until this value is known until it's actually been retrieved this is how polumi allows you to do things in a specific order um it allows you to do things in this way because it it it will will grab this vpc value here and then pass it to this resource here and because it's passing an output from one resource to an input of another resource pollumi knows the ardency and it creates this dependency graph for you does that make sense it does yeah so it goes through and determines the ordering and which things require data from which other things exactly yes and then the final thing which i think is interesting especially if you've written python before is that we're actually using python's type system here um in order to actually make our life a little bit easier so this ingress property could just be a dictionary right like it could just be an unstructured dictionary but then you would have to look up what would be need to be in that dictionary right like you'd have to look at what the protocol was what the front part was what the cider blocks were um and so we have added this type system to our properties which is uh some of these names can get super super long um i'm not gonna lie like they can often look a little bit um unwieldy but what it means is that when you're actually authoring this stuff it means that if you get anything wrong like let's say for example i wanted to set the protocol and of course that's a bad example let's do let's say i wanted to set the protocol to true i can actually examine this and it will say the input should be a string here you'll notice that how it's hovering over this is not a valid input for the actual paloomi program and so by having this type system available to us um it helps us know when we're authoring our program um what the correct inputs and outputs should be does that make sense it does yeah and you can even set up automated linting such that you could check all your types when you're uh yeah i have i have been struggling with my my pi implementation because i've got so many virtual ends that it's actually not working for a change um but usually if i set this to true um on there like it's gonna actually say here that it's not defined um but if i usually what would happen is it would tell me that that's not a valid input for that property if you think about how that's how that looks in other kind of especially in a yaml kind of mechanism you would actually actually you would actually have to provision the resources and get the result back from the aws aws api before you knew you'd made a mistake right like yeah and so it really does help you with the feedback loop of getting things correct um while you're in the authoring process got it and we've got one question from the audience uh a little bit nuanced around the state file and where that vpc.id value actually gets stored so when we're running the pollumi code presumably it's it's getting loaded into memory but does that also get persisted into the state file it does indeed yes so like once we've run this get vpc call here and once we've created these two this is called a function in polumi it's a a retrieval mechanism but the value that it gets returned all these different outputs that we saw from our vpc they all get stored in state right so they'll be you'll be able to use those and reference them to your heart's content and then also when we create this security group it's going to have those same outputs it's going to have the security group id and the security group name all of those get started in state two and you can also reference them across pollumi programs using something called a stack reference um so all of that does get started state that's correct cool okay i'm gonna do some more copy and pasting now sid because we're uh we're um we're on we're on time crunch um so i'm gonna set up a load balancer for our aws uh fargate application very similar to what we did before we've used an string for the name we're passing the security group that we created here to the security groups for the load balancer and we're also passing the subnet ids so this load balance is not going to get created until the security group id is known and the subnet id is known so again we're creating quite an uh quite a complex graph here um and then the next thing i'm going to create is a target group if you are following along at home i've checked this all into github i'll share the link out into the comments afterwards so don't worry if you i did write this once um and so i don't want you to think that i'm just kind of like copy and paste for my living although that is kind of what i do um so then we're also going to create a target group um for our um fargate application we're going to make sure that every application listens on port 80 because that's generally what we want our web applications to look look at uh we're going to say it's a http protocol and then a target type of ip and then the vpc id is again coming out from our vpc call um you can you can see how easy this gets if you uh if i fill this in myself so if else dot vbc dot id like so and if i try to set the arn here if my my pi was working correctly um it would tell me that that's not a valid input and i'm not allowed to do that right um okay a couple of other resources we're going to create we're going to create a role uh an iam role so that our application can actually run inside fargate you'll notice here that i'm actually using the json package from directly within python and i haven't imported it so let's go ahead and do that import json which shows you that within your pollumi programs you can use pretty much any python package you want so if you have a favorite python package that you want to use in your pollumi program it's totally possible to do so i'm using the json package to help me create my assume roll policy right like it's it's gonna it's gonna turn this dictionary here into um json that gets passed to this property here um which i think is uh often a really really nice way of being able to create things right yeah for sure um final few final few things that i need to create here so i'm going to create a role policy attachment which allows my um i am role to actually run ecs tasks within fargate um the rest of it's pretty self-explanatory i'm gonna make sure it's attached to the role itself and then i'm gonna get to the real the stuff that actually matters um and so i'm gonna copy and paste this last one and then we'll set up the service uh we'll type that out so what i'm going to do is create this task definition which is going to actually run our ecs task within aws and again because we're defining best practices for our end users here i'm going to set some values for them that we could potentially make as arguments later so i'm going to say that we're going to have a minimum cpu and a minimum memory of 256 and 512 and then if you take a look here this is where our arguments come into play right so i know that i want to run a specific image as part of my web application so i can pass this argument that i defined all the way up here this image excuse me this image argument and i could pass this to a property like so um so i can say my at my image that i'm going to run is going to be here what we could do if we really wanted to is we could say something like we want to create a port argument and let people run um you know applications on whatever port they want and pass it down through all of these different properties but because we are again where the we're the experts as the devops engineers so we're going to actually define best practices for them right yeah or we could add things like the resource constraints if we wanted to specify cpu and memory uh we could vary that across different stacks right if we had a dev and a production stack exactly that's exactly right yes um and so now i'm going to set up the actual service that's going to run this particular task definition and it's going to be similar to what we did earlier so i'm going to create a service like so and i'm going to use an f string to create the name that's not the correct way to do it so like so service uh name see sometimes your ide knows more about how to code than you do i'm fairly sure at some point in the future we'll be just letting uh we'll be letting uh code ai right as far as um so i'm gonna pass all of the properties that i need to actually define for my service here and what i can do is i can actually look at uh by using go to definition inside um inside my ide i can look at all the different things all the different inputs that need to be defined inside here right so you'll see lots of these different things are optional and sometimes they aren't um but i know because i've already run this before all the different inputs that i actually need to just to set here um and so i'm gonna set a desired count of three like so um i know i want my launch tab to type to be fargate because we are obviously defining best practices for our users my task definition i already created earlier so task definition dot arn and of course i get all of the benefits of my uh output completion my intellisense uh so these are all the different things that i could pass from my task definition i'm gonna pass my irn here um set up some network configuration um which is where we're of course going to use the strongly typed um inputs so i'm going to do service network configuration ads and then you should see that i can look at all the different inputs here like i know that i want to assign a public ip so that it's actually going to run properly um and then i also need to define define some security groups and some subnets um by using this strong typing inside the python and it's not the default in python right like this is not necessarily if you've written python before you might not be used to strongly type python um but i personally really like it i think it makes your life way easier um to actually um you know to actually get things running yeah i think there's there's a little bit of work up front to get all those types in place but once you have them and you can leverage them it makes both maintenance and writing new code much much easier i totally agree um and so then what we need to do for our um service is define a load balancer which of course we already defined up here in our polumi code so what i'll need to do is do aws dot ecs dot service load balancer args like so and then we're going to specify the target group arn self dot target dot container name equals the name of our running container and then we also know the part is going to be port 80 like so um and then the final thing that i'm going to do here which i think is an interesting thing is i'm going to specify some options but i'm not going to specify just the parent right i'm going to specify more stuff here that i think is interesting so we have this depends on input as a result option right and what we're going to say is that we want to make sure that this service doesn't get created until the listener and the task definition have been created why have i done that right well the first reason is that um this actual uh cert this service mechanism here doesn't actually have an output from this listener we created right like there's no there's no need in aws to create an aws listener wherever i've put it um where is it it's not consuming any data directly so your your graph dependency resolution can't figure out the the dependency and so you're explicitly calling that depends on so that it can add that and sequence them properly that's exactly right yes and i've just actually realized that in my copy and pasting frenzy that i didn't actually delete add the listener so this is a nice exercise in live coding so i'm gonna add this here myself.listener um so because i'm not passing any of the outputs from this property to this resource here i can use depends on in pollumi to make sure that it will actually create things in the right order right um which i think is a really interesting mechanism because often there'll be times when you're creating infrastructure code where you want to do things in the right order at the end of the day right um so this is this is a fully uh defined reusable component in pollumi um and so we've done everything here that we need to do in order to actually um in order to actually create a reusable component in a traditional infrastructures code workflow we would re we would use this in our pollumi program like so we would do import web app like so and then we would create app equals web app dot web app like so because that's obviously the that's the reusable instance that's the reusable instance here right and then we need to specify a name i'm going to call it example and then we would do aggs equals web app dot web app args image equals nginx like so right and i can obviously create lots and lots of these right i can do app one equals example that one this is the thing for me that started to make sense when it came to infrastructure as code right like i am going to be able sorry with with object-oriented programming i'm going to be able to reuse this thing over and over and over again and i can set all these different properties and i can do this as many times as i want this is where i personally started to kind of get an understanding of what object orientation was because i was applying it to something that i already knew i knew that i was going to run lots of different web apps in my fargate instance and this is my implementation here right yeah the other interesting thing right is we could have an array of example and example one and then map over that or do a for loop over that do a for loop over it and create a new web app for every single one absolutely definitely um you you know this is really hard to do in other infrastructures code languages or anything like that um that see this is a great use case um of being able to do yes that's exactly what it would look like and i use this all the time in most instantiations simply because it makes your life a lot easier um and so what we would normally do here is we would run pollumi up like you might put this in a ci and cd pipeline right like you might um you know do this um over and over and over again in a cicd pipeline and if you um if we go back to our code real real real quick remember here how i set this token and i was like this isn't going to make sense until i run my paloomy program this is where this ends up right so this ends up as like a unique identifier within the actual pollumi state itself right um and there's a couple things that i think are interesting um i've set the parent property on a bunch of these different resources what that shows you in pollumi is it shows you this nice dependency implementation here so you can actually understand which thing relates to each other um which i think is nice um and this would work great right this would be a totally fine implementation um and lots and lots of users uh use pollumi in this way however there is another way that you can actually do this and it's without using the paloomi cli as an in as an implementation right so what you can do is use the automation api to create your own kind of running mechanism um and so what we're going to do is we're going to use python's arc pass to create a little custom um a little custom cli tool that we might pat might hand to our software developers in order to run their infrastructure so i'm going to do import pass because i know i'm going to want to use that eventually and i'm also going to use import sys like so um and then what i'm going to do is i'm going to create a function called polumi program and i just want to call out one thing before we do this if you want to go back to the webapp.pi where we actually are calling using that or sorry main.pi is fine but the image name we had before was just nginx and so that represents the actual web application that our development team has built we're just using the the default official nginx image uh i just want to make sure that anyone in the audience who has maybe missed that leap is like that would sure yeah that's we would have our own custom image there that would have our custom web app and that is what would get used uh within this fargate uh deployment yeah i mean what you could do is you could actually build the docker image locally as well i can actually do it as a full one-stop shop um i'm just using nginx because i know that off the top of my head as an image name which i think is uh kind of indicative of how many times i've deployed that image um so what i'm doing here is i'm defining a function that's going to be our paloomy program rather than writing it directly in my main.pi right so i'm going to do the very similar to what i did before i'm going to specify the image equals nginx like so and i'm going to make this configurable shortly um but i think it's uh important to know that i'm just going to use this nginx image for the time being and then i'm going to do pollumi.export i'm going to export the url the dns name so that i know how to get to my my running web application right that's coming from our load balancer resource that's right yeah um and i haven't defined my name yet so i want to make my name configurable so why don't we do that using python's arc par so i'm going to do argpass.argumentparser like so run a web application like so and then i'm gonna do parser dot add argument name metavar equals name this needs to be a string and help is going to be the name of your running web application like so like so and then we're going to find a final then we're going to find another argument that is going to be an optional argument that is going to allow us to destroy this particular deployment once we've finished right and so i'm going to do action equals star true and the default is false because we don't want to obviously destroy our application each time and again if you think about how this looks in kind of a real world example it allows you to really um kind of define um your own workflows within your organization you'll know better than anybody how people get things done inside your organization and if you can create little command line tools that will like integrate with your ci cd pipelines or maybe integrate with your authentication mechanisms or integrate with your kubernetes clusters with all the different inputs that you need this is just one example of how this might look right yeah and all this argument parsing stuff is just vanilla python um just standard off-the-shelf python it's a i'm not even gonna have to install this as a dependency because i believe it's part of the python 3 standard library at this point right um okay um so now i'm going to define a couple of variables i'm going to say project name equals i'm always going to create this in a project called web app and then because i made my reusable instance of this um a flag like an argument flag here i'm going to call my stack name equals name and then i can start using the prelumi automation api to actually start creating infrastructure so what i'll do is i'll create a pollumi stack and i'll do pollumi.automation dot create or select stack the stack name is going to be stack name because we're gonna make that configurable each time and then the project is going to be project name and then the program is going to be of course this paloomy program right like we've already defined our pollumi program as a function here so i can pass this palomi program from one place to another if you think about what that's actually going to do it's basically automating the pollumi up portion of all of our created infrastructure right um which i think is um you know considering how that might look for most people um you know not having to do a bunch of yaml templates in a um in a ci cd pipeline writing this as kind of actual application code or actual python code is just a really nice mechanism i think and then for anyone who wants to actually execute this cli will they will have already needed to run paloomi login or equivalent on their system they already need to actually have a place to store a star state okay the automation api is uh i believe is going to have or is uh soon going to have a mechanism that will allow you to do a login um so you'll just need a place to have stored your state um and then once you've done that um you can actually create things to your heart's content basically got it and then we are defining name via this argument we are not passing that name into our paloomi program function yet do we need to refactor we are what we're doing what we're doing is for each instantiation of a polumi program we're going to use this argument name here as the stack name which will mean that we can run this over and over and over again so i'll be able to do python main dot pi name equals my cool app and then i'll be able to do python example my second cool app and it will create a new stack for each of those right like so that's how we're using the name at the moment my question is how does the name on line nine get uh defined um it's from this argument from this arc pass so it's here so uh args.name oh got it got it i yeah sorry and then it's just included via closure yes exactly that's exactly right yes uh i didn't actually know the nomenclature one of the great things about working at palumi is you use different languages and they all have different nomenclature define the same thing and i'm like what is it called in python and apparent it's called a closure um so um i'm gonna do a little bit of copy and pasting because i am um aware of the time uh so what i'm gonna do here is if you remember when we created our paloomy program we did a pollumi it was it asked us for the aws region because it's a required configuration value so i'm gonna set my us2 region here um as a set config and then i am finally going to do a print statement that will do a stack dot up and then print all of the information to screen right so this is the finished um like the finished implementation of a uh automation api um mechanism right um so what we can do now is we can do parser dot add argument image metavar equals image like so type equals string help equals the image you'd like to run like so um and so now we can make that kind of optional currently it's hardcoded to args.nginx so i can do ash.image like so um and now i can specify my own image whenever i'm running my python automation api program makes sense it does yeah and if we wanted to have arguments to palumi program that function could right now we don't we're passing everything via this these global args.image and args.name could we have arguments to that function and pass them when we call that create or select stack somehow absolutely yeah you could do image string like so and then you could do um wherever my pollumi program is and then we could do uh args.image like so yeah absolutely that that's that's a totally valid way of of kind of doing it as well um if you really wanted to um for the sake of this particular um for the sake of this particular um example uh i've kept it nice and simple um but yes like you have all of the full paradigm full paradigm of your um language at your disposal at the end of the day right um so you you can kind of do whatever you really want and kind of lay this out how you however you want i've kept this really really simple because we are obviously only half an hour this could be a completely elaborate thing and i've seen people do web applications with the automation api behind flask and all kinds of different implementation mechanisms cool yeah and the yeah i just wasn't sure because we're passing a reference to the function versus calling the function in that creator select stack so i'm sure there's i can look at the docs afterwards to see how exactly it would handle it but just wanted to make sure that you could pass arguments there so if you look at this if you look at this input for this program it takes a paloomy function which is a special type of function which is a pallumi program but this is just a normal python function right like so you can use these functions in whatever way you want you can define input arguments you can set defaults you can do all kinds of different things there cool okay so let's try run our python application so let's do python main.pi and we should see that we need um to we get some help um this is where they kind of use like the user experience comes in with using the automation api and again you've seen people do this with flask i've seen people do this with all different types of things but of course we need to actually specify a name and an image for our actual run so let's call this uh sid i'm going to run a sid application and then of course i'm going to run nginx i could of course run multiple different um you know multiple different uh images and so i can also specify help and you'll see that i have this optional destroy flag here which i haven't currently implemented so let's take a look what happens when i run this the moment of truth yeah you see all my print you would see all my print statements are kind of going ahead here as well um so it's going to do a bunch of installing plug-ins it's going to refresh the stack and then you can see my pollumi program is being executed in much the same way as it would have been if i was using pollumi up but the user experience here for anybody downstream using this kind of component that we defined is just way better all they need to do is specify a name and an image they don't need to write a single line of code if you think about how this is generally used this is generally defined by some best practice experts you hand it over to your dimension users and say you want to deploy your application to our infrastructure to our well-defined um platform here use this python application or use this go binary that i created or you know use this javascript package that i used um the user experience is kind of way different to like kind of shoving things in ci pipelines you can of course use this in a ci pipeline if you want but all you would need to do is just kind of define things as like this and of course you can package this up as like a standard python cli application i'm just using the python interpreter at a moment but this is just a completely different mechanism in order to do this yeah and if people are interested in so we're just using the standard library arg parse but there's a package called click which is really nice for building clis that could be used as well yeah i think they use these decorators right on the different phone like on the different uh stuff right yeah i think so and you'll see like when we've created our web app you'll notice that the name sid is being used here as a stack name right so what i can then do and this is really useful for like multi-tenant environments like if you if like lots of people are creating lots of different um lots of different infrastructure that's kind of similar but not quite the same stacks are really useful here so i could have my own instantiation of this you could have yours you know you could have a test instantiation which is kind of uh you know defined elsewhere um and then all we're really waiting for now is we're just waiting for um all the different parts of the aws um infrastructure to be created so it'll take a little while to create a task definition it'll create a little while to create the load balancer you'll see that load balance is creating those counts sometimes take a few moments in aws because they uh they they provision paloomy's pulling the uh load balancer creation and when it's finished it will come back saying it's created um like so and then i think you logged out the the url as well at the end right so that's what we'll get and once we're done and here's my here's my low balance url what you can do is you can create and i thought about doing this for this particular uh demo as well you can use something like rich to actually make way better outputs i'm just using a super verbose output here but you can use something like rich the python package which i think is really awesome um python rich this is an off awesome rich text library that will allow you to actually define way fancier output mechanisms uh you could do something like uh create a spinner create a progress bar um and then output the actual uh url like so is it gonna work let's give me a https of course uh and of course i'm getting a service temporarily available because you know live things never work the way you want them to um let's try again there we are i was trying i was trying https instead of http um i think that's chrome uh telling me whatever so that's my nginx uh deployed um and of course i can completely i can create my own instantiation of this like so um just call it and create an l brig stack and it's going to create the whole thing again from scratch with a new low balancer and all that kind of stuff and it's as easy as running a command line tool super super simple and straightforward awesome yeah let me check the chat here we've got a question where can you see the resource arguments for the aws resources um so there's two places you can see them um we have the pollumi.com so if you are used to looking things up on a um on a web application you can see all the api docs here so all these are all the different modules we have been using um where is it uh ec2 so we created a vpc so all these are all the different inputs like so um so here's all the different inputs that you can use like uh tags and all that kind of stuff for a new vpc that's one way to do it the other way to do it is to similar to what we did before is to actually start defining your paloomy program and then just go to definition and actually look here so all these are all the inputs and properties that you can see inside a cluster class for example i generally tend to uh use um a combination of both so i'll have i'll look things up here inside the actual api docs which is under pollumi.com for slash registry packages um and you can start drilling down into all the different things you create so like if you're an azure user if you look at the api docs these are all the different things that you can create and we have this awesome uh we have this awesome language picker on these different resources um and so you can see how things look between python go c-sharp and typescript um so if you use if you're using any of those languages you can see examples in how to do uh most of those things from there as well nice nice um yeah i think this this gives a really good example of allowing the infrastructure team to encapsulate uh or hide a lot of the implementation details from the development team if they don't need to know them so you can expose what they do need to know about and what they do need to configure without needing to dive into the the weeds of uh all the specifics of which infrastructure is code tool and how things are are operating under the hood and i i've seen this in it i've seen this in practice right like i've asked people to write you know all the different infrastructures code tools and be like it's super easy just look at this example um and i've asked you know developers have said hey i need to deploy my sample application and we're like well you've got to write a bunch of json or yaml or other kind of dsls um and they'll take one look at this and they're like i don't even know what a security group is like i don't know what a security group looks like and in my experience excuse me in my experience handing them something that's already defined this even if it's as simple as just the actual python package which of course could be um published to pip or if you're using npm you could publish it as an npm package and saying hey just do an npm install or a pip install and then use the actual package like that it's just way more intuitive to application developers and then the final thing that i'll say on that note is that it's often the case or it certainly has been in my career in which there's far fewer devops engineers in an organization than there are application developers getting them on your side and getting them the deploying that deploying their own applications really helps break down those silos and means that they don't have to create a ticket to get you to help them it really reduces bottlenecks within organization uh you know within organizations we have some large companies using pollumi at the moment and they will you know if you look at some of our case studies uh a great example is snowflake who are huge huge uh polumi uses for our go sdk it's really completely invigorated their engineering organization because they can just use the languages that they're already familiar with and i think it's really a testament to how important the input the input and authoring experience is for those organizations it's such an important part of it yeah and i think the the automation api is not going to be right for everyone right some organizations have lots of people doing infrastructure work and modifying things and are familiar with the tools but once you get to the point where you have a mature enough platform you can build this and and abstract away some of those some of the nitty gritty yeah i think a lot of a lot of users uh polluting users start just using pollumiop and writing pollumi programs in a kind of ci cd pipeline way and then once they kind of migrate into that platform engineering type uh path the automation api really kind of takes on a life of its own and becomes super super important awesome well i don't uh we've got one more one more question here in the chat can you use palumi to configure like ansible uh no so polumi um operates in an api layer um and so um with polumi you generally talk to an api and it provisions something like that so if you have something within your organization that um you know kind of configures an operating system with an api then you're able to do that what we generally see is the ansible and python sorry ansible and pollumi go together whether it's putting your ansible um inside your user data um or um configuring things um you know uh the we have something called dynamic providers which allow you to actually insert things into the pollumi life cycle um often users will use um ansible at that stage there is this really great um um the mit folks are using polumi um and i can't remember the uh uh mit dl i think it is uh of course i can't remember off the top of my head uh the the uh mit folks are using palumi and pie infra um to actually um kind of fully manage their um uh the full life cycle of their uh here i've found it on my other browser so this is an instant an example of using something similar to ansible pi infra and uh polumi uh to actually provision full uh a full learning um you know a full learning example in python um so you can see here we've got a paloomy program and then they use pi infra to actually do the operating system configuration at the vm level which would be similar to uh ansible in that level does that make sense yeah i've seen that pattern work well where you have a tool like pallumi handling the provisioning aspect and then you have a tool like ansible handling the configuration management that's right yep awesome well thanks so much lee for coming on today uh thanks for indulging me i really appreciate it yeah this was great um i think we we made it pretty far we did some copying and pasting to speed things up along the way but we built a a simple application uh deployment platform in just about an hour i'll i'll hop in the comments with some links once we finished kind of shutting the live stream down but i really appreciate you having me uh sid i think it's uh it's always fun to kind of kind of build something uh together and uh hopefully you uh hopefully everybody who's watching along kind of saw some of the power of using these real languages to define things with your infrastructure perfect with that we'll sign off thanks so much take care everyone

Channel: DevOps Directive

Views: 449

Keywords: pulumi, paas, heroku, aws, devops

Id: 3a23l8rfJJI

Length: 67min 32sec (4052 seconds)

Published: Tue Nov 16 2021