AWS Dev Day Australia 2018 - Building a Secure Cross-Account CI/CD Pipeline

hey guys thanks so much for coming this afternoon I know it's been a long day for some of you really appreciate you coming to joint I think this is really interesting an important one which is why I build to talk about it so hopefully I can help you out with wherever you are at your journey on the multi-account adventure so basically when you start out in a in an enterprise you'll find that these companies are at different levels of maturity with how they organize and choreograph different teams and different accounts with their cloud footprint and so what we're saying is that there's there's a lot of confusion about how best to do this and so over the last sort of two or three years we've had a lot of work done globally on how we can come together and build up some best practices and advice on how you can set things up the right way so that's what we're going to cover up in this session I'm going to run you through some best practices on setting up your deployment processes and DevOps tooling across accounts and I'm going to run you through a little demo that we've got to show you kind of how it works mechanically so what we do see this conversation often arising from is it comes from challenges that people are having and the first one of these is generally coordination across teams so when you're in a big enterprise complex environment particularly coordinating your applications that are connecting to other applications in the company is complicated and then working with the teams is hard coordination of deployments getting to rolled out coordinating with other teams on time slots all of this stuff is quite challenging and this is something that still happens in the cloud if you don't orchestrate these things the right sort of way there's also environmental complexity that comes to play so when you are when you've got these different systems that are interconnected you don't want to break other people's systems or interfaces with your deployments and so orchestration becomes really challenging and change can probe processes have a tendency to slow things down and so in the modern world of agile software development we're trying to find where you can take out some of the blockers and having multiple accounts is a way of segregating things and ensuring there's a security blast radius that's contained if there's ever a compromise but at the same time if you need to push something out to your dev environment and then you promote it in to your test environment and then you've got multiple test environments and then you've got multiple teams interfacing with those test environments this becomes really hard and it can slow things down a lot so these are some of the challenges that where we see quite commonly so when you're starting out it usually starts with a proof-of-concept or a small project and it might be one dev ops engineer or developer that just starts building stuff in the cloud and that generally follows if it's successful with adding a couple more environments so usually what happens we see people spin up another V PC within the same account as the existing dev V PC and then for production there we see people really commonly making a good decision which is to isolate that production environment is its own separate account so that's good then once companies start maturing their practices generally we see people trying to get some sort of isolation you you've got a little bit of maturity in the way you're doing your terraform or your cloud formation you start breaking things out and you realize that you can stamp out these different VP C's into different accounts pretty easily that's a direction that we're seeing things sort of trending but what do you do with the pipeline's what do you do with deployment well one thing we see quite commonly is the pipeline's getting set up within each of the different environments that you've got running in the cloud and that's that's commonly done because as an engineer you're generally just trying to get something done to get your your dev environment automated so you know doing all the manual processes and that's a good intent but then when you want to roll that out in other environments often people will take those same sort of scripts or configs and then push them into those other environments problem with this is configuration drift so if you define a pipeline you've got a of different steps in there but you've spent a whole bunch of time defining really carefully and it works really well in that environment and then you push it into a different environment all of the time there's a bit less diligence over making sure that it's consistently applied in those environments so the scripts tend to take a path of their own even if you are using infrastructures code to define these pipelines so there's a bit of a risk there that pushing those pipelines into their own different environments is going to cause your configuration to drift and you'll you're going to lose control over making sure that with the pattern that you used to deploy a plot to deploy in per dev environment is exactly the same as your test environment is exactly the same as your prada environment so that's one of the the challenges we see is very very common the multiple pipelines pattern so iterating on that some people see this issue and will build a single pipeline that spans across the accounts now this is a great thing to do because what it means is that in a single view you can see which exact phase in your deployment pipeline code is getting promoted through so if it's getting pushed to your test environment you can see that what steps it needs to pass to get to prod so that whole process is visible and it's also good because you're not going to have configuration drift to the same extent you can make sure that the pattern that you're using to promote to dev is applied consistently to test and deployed so that's a that's a good thing now the challenge with this approach is that you are as you can see there you are you are hosting your pipeline in one of those three environments so you kind of be kind of coupling your pattern really the configuration of how you manage your change which is what a deployment is it's just change management you're you're pushing that into into the context of your application so if it's hosted in your dev environment what does that mean about your security profile when you're pushing stuff from dev into prod yeah I would say it's problematic so what we're starting to see more and more customers do is build out a pattern that looks a little bit more like this so you've got in this one you can see an account that's completely dedicated to the DevOps processes and tooling and then the application environments have accounts of their own and this is a great pattern that we're seeing a lot of success with because it's keeping everything separate and isolated and well-defined so what you can do with this is you can define exactly what security parameters are required to push stuff into your dev environment and if you want those security parameters to be to be different for test or prod for any other reason you've got full control over that but this is a visit this is a good way forward and the other thing is it doesn't need to be just your DevOps pipeline any other DevOps tooling that you use across different environments can have a similar pattern apply it for example one thing that we see some people doing is a particularly newer cloud native businesses is using tools like the simian army for those of you that haven't heard of it chaos monkey is a really famous tool for doing chaos engineering which is introducing randomness into an environment to test out the durability of infrastructure so you could for example run chaos monkey script in your in your dev environment it from you can launch them from your DevOps environment push them into dev and then if you're comfortable that the way that you've got your architectures configured in dev is durable and by trying to randomly shut down servers or kill networking connections it will continually bring itself back alive it'll self heal then you can push that to test and try that out an ultimate would be running your chaos engineering practices in prod which is making sure that you are super confident that your your production environment is alive and durable and happy so that's but that's great that's for a single team for a single application but how do you handle that when things get a bit bigger well that's what that's what we've seen more and more people trying to achieve rolling out patterns like this at scale and this diagram captures a really good way of achieving that so you can see that we've got these enterprise white accounts now these will generally be managed by say your cloud center of excellence and the CEO II might have they might look after for example connectivity networking account and shared services but you might have InfoSec managing their own security account that they use and those tools might pull data from different accounts all across the organization for example so the enterprise wide accounts are generally centrally managed either central IT cloud Center of Excellence InfoSec teams like that the developer counts as something that can be provisioned out for everybody as sandboxes and you can use policies to define what those accounts would look like touch on that's surely and the business unit accounts is where you've got the the pattern that I just described before so that's where you'd have your DevOps tooling and dev test prod for each application and you might use those accounts for multiple applications but within single business unit so you know the marketing team might have their owns that marketing engineering or your operations engineering might have their own pattern like this so this is happening because customers are getting to the point where they're not just spinning up your three four accounts this we're seeing people spend up tens dozens and even hundreds so you need some way to plan for this and to manage it and we're seeing a lot of success with this and make some things a lot simpler but the big question is how do you actually achieve it so to be able to do something like this as a service that we've got that was released a while ago now called AWS organizations and organizations helps you deploy out multiple accounts and it gives you the option to use policies to define exactly what is enabled and what is not enabled in those accounts so the first one of these is the ability to control those AWS services so you can say that in this account elasticsearch is available these set of accounts we've got a different policy so for example we want to disable elasticsearch to be deployed out across all of the developer sandboxes but you can choose for your for the business unit V PC VP season accounts you can say that you know everything is available but you can have the central IT reporting back to what sort of services are being spun up as well and you can mandate that certain services are available or not really important feature is the ability to automate account creation so that's one thing that organizations does which is pretty critical if you're going to be doing the developer sandbox is at scale really important one and then the ability to do consolidated billing and usage reporting as well so this area is is complex and it becomes more complex trying to assimilate all of that information as scale so having something like AWS organizations is very important for that purpose so there's two ways that you can about go about actually technically doing this so organizations is the first tool that you will use kind of no matter what because that helps with the account part but AWS landing zones is something that we released just a couple of months ago and the landing zone is a as a solution it's a pattern and it's also a cloud formation template that lets you get started if you do not have a big multi account pattern already in your organization so landing zones is basically an easy deploy easy to deploy solution but it just sets you up so that you can do all of that automated account provisioning it's based on best practices that we've seen this has been built up over the last two or three years so it's not just AWS kind of making it up like most of our services it's stuff that we've had a lot of demand for over time and we've taken that on and we've learned from what does and doesn't work and we've set this up to try and guide guide you and give you guardrails to make sure that you don't make mistakes that others have before it gives you the initial security setup and governance controls that you need so it gives for example one of the important parts that's listed there is automated deployment but it sets up all of the governance around how you can make define your policies that are used to build out these other accounts Electric baseline accounts and it gives you that that concept of an account vending machine where you can just turn them out as required and you've got control so that you can spin them down as well so this is a really good starting point for implementing a multi account strategy so you can see here that we've got a couple of different accounts so the organization's account parent one is there at the top there's a shared services account a logging account and a security account and we've got a bunch of tools in there but I won't run through in detail they should give you a feel for you know the different services that are deployed automatically as part of this to help solve the key problems that we see people having when setting these things up themselves so this is publicly available you can just jump on and download those scripts and it'll give you a really good head start but if you're already in a position where your organization has a bunch of accounts already at play and you need to start sort of wrangling them together and it's exercising a bit of control over that then the other option is just to roll your own so we've got a bunch of different tools available as you would well know I'm sure code pipelines what we use to orchestrate a lot of the different elements of the DevOps tool chain and then of course you can roll any of your own from marketplace and there's a wide range of those available now one important tool that is going to play a key role in doing this stuff is Systems Manager now Systems Manager is really great for orchestrating a whole bunch of different activities across a server fleet but for the purposes of this session we're just going to have a look at parameter store now parameters store is something that helps you apply the 12 factor best practice of separate separating out your code from your config so you can use that to store your secrets and if you've got connection strings and things like that that you need to store somewhere then see parameter store will do that job now my friend aurelion and paul have a session after this if you want to find out more about secrets manager which is another option that you've got available then they're going to be running through that they're using both parameter store as well as secrets manager but essentially for our purposes it's just a very simple place for you to store secrets that you can then use as part of your overall multi account setup so a little demo that I wanted to run you through is called docker invasion so there's a QR code there you can go and hit that right now if you like and you can have a quick go now it uses two micro services we've got ECS and we've got lambda the game itself is running in a container docker container and the topstick or service is running in louder so though I was just going to run you through what the flow looks like for those two different scenarios quickly so for the game service itself the way that this works is a developer will check in the code to code commit and then that will get picked up by code pipeline code pipeline will then trigger code billed as the next stage code build will build the docker container based on the new code that you've checked in and it will store that container in our elastic container registry so I know that we've had a few docker sessions today different container sessions I'm sure this has come up ECI uses s3 it stores your information it stores the build artifact in there the docker container and the way that it works and this is the key part code pipeline or whatever your tool chain is your pipeline tool will assume a cross account roll into using iam so when it does that it will then push the cloud formation script in that CloudFormation script has the details of the difference in ECR of what container it needs to spin up ECS will then get that change will get pushed to ECS which will then reach out to ECR and then it will pull that pull that container out and deploy it onto the ec2 instance that the container is running on so that's the the game service now at this point we've just deployed to dev and that's great but we probably don't want to just ship stuff out the door to production so what you need to do is make sure that you've got a gate setup to ensure it doesn't just flow straight through whether there's passes or fails of tests so there's a few different options one that's commonly used our black days where you have as a gate that says do not continuously deploy during these particular time windows that's often if you're like a software-as-a-service retailer like online retailer you might want to not deploy during 9:00 a.m. and 10:00 a.m. each morning for example other options are you can gate it by just having a simple manual approval step and that's what we've got in our demo and then a few other ways that you can do that as well but the important thing is that you've got the exact same pattern happening where a cross account role is assumed into production and then it will push the same pattern through deploy the cloud formation script retrieve that build artifact and then push that out to ECS and easy to so that's the game service top-scorer service we've got is a little bit different so similar pattern in that you check your code in to get get that picked up by the build pipeline that then processes that and writes the artifact to your s3 now in our case that build artifact is a lambda function and the lambda function get needs to get pushed into those two environments so code pipeline will then assume across account role make the change to cloud formation push that in cloud formation will then retrieve that build artifact and push it to lambda so in our case what we're doing at what we wanted to show you was how lambda will use a different connection string for the DynamoDB endpoint in dev to the DynamoDB endpoint in prod so it will retrieve that from parameter store that endpoint same thing happens when you're going into production of course you want to have some sort of gate in the way but as long as the gates there you can assume the cross account role and then push it into cloud formation retrieve the right parameter when the function is executing so I am I've recorded a little view to show you exactly how this works we switch across please thank you okay so what I'm going to show you here is the app itself for those of you that scan the QR code you'll see this it's a little space invaders clone OOP you okay and so from there we're going to jump straight across into code pipeline in the console code pipeline we've got a few different pipelines there so the one that we're looking at as I said before is the EC s1 so we've got a source stage where we will get picked up when we commit stuff got a build stage where we're using AWS code build we've got a deployed dev stage and deployed a prod stage so you can see in there that's where we've got the manual approval listed so that's our gate that we're using to make sure that we get the right thing into brought into prod so this is the code that's running that we're going to be running in our container so you can see in here I'm just it's just a little JavaScript app so I'm just commenting out the background color and putting it to something a little bit more interesting so we're just going to add that to the change and commit it bring the party now we're just going to push that in to the repo so you can see this this is my internal private code commit repository okay so now that that's being pushed that's going to get picked up here in the pipeline automatically so you can see that it's been picked up at the code commit stage and processed now while the build stage is occurring I'm just going to show you what that looks like that that the build definition so this is code build and you can see we've got the repo listed there and if you click on the project details the build spec is listed so that build spec is defined in llamo as part of the project so you can see in here we've got the docker build stage and then a step to tag that build correctly and that's tagging with what we're going to use for in ECR and then the docker push pushes that into into ACR so that means that the container once it finishes building will be stored in s3 there so once that build finishes it's going to deploy that automatically to dev and then once that's done it's gated by that manual approval step that we've got in there so the manual approval is using work mail so this is a one oh this is our managed email service and you can see that it sent us through an email with the content to review which is the staged version of our deploy and you can see the backgrounds colors changed is something a little bit more body so to approve it you just click the other link that pipes you into code pipeline again and it will come up asking you to approve or reject the revision so this is the manual approval step to make sure that the right content is getting through to production so this is an excellent pattern and as I was saying before you don't need to use code pipeline all the commonly used DevOps tool chains will support these these features so now that that's been manually approved that will get pushed to production and now we just do a hot refresh on that page and you can see the colors change in pride so that's the Fleur for the game service itself so the other one that I wanted to show you is the the lamda change that we made as well so we just used to put this together we just use the sample code that was available it's got a source and a build stage similar sort of pattern now this time we're creating a change set with cloud formation so that we can check and validate what change is exactly going out there's a manual approval stage and then the same pattern again so you can see exactly what the different stages are and again having this all in one pipeline makes it very easy to avoid configuration drift so just looking at the build stage I'm just going to edit that project and have a look at the build spec so you can see here in the build stage we've got the the command in there to package up the cloud formation script and store that in the in s3 so this is the actual lambda code that we've got in there and as I said the reason I wanted to show you this is so you can see how to assume the cross account role from the context of the lambda function so that's what it is it's STS dot assume role and then the response that's hitting the systems manager and using the get parameter command and that's passing that into the dynamodb call so that is how the that's that's how the lambda function works to do the cross the cow stuff so you can see that both those are pretty straightforward if you build them and you use the right sort of patterns it's not too hard to do you can set it up so that cross account stuff becomes more of a non-issue but it's very important to have the structure set up correctly so as part of the conversation when you're getting this done you need to include all of the different dev teams in this journey ops teams that are involved that's very common particularly the big enterprise like complex enterprise environments any DevOps teams your cloud center of excellence and InfoSec as well these all need to be part of the journey when you're setting up like proper multi account automation because a lot of people will have a lot of opinions on how best to do automation so you need to get on the same page you can't just beat people into what you say tell them is best so this is a team effort use a separate account for the develops tooling that's a really good pattern as I said before keep things separate define the security context dev is going to have a bunch of engineers usually with a lot more lack security doing all sorts of crazy stuff you don't want that security context to be applied to your pipeline which can then push stuff into production very bad idea use cross account roles to coordinate your administration of these things and the different deployments and a key tenet to this as well is to build once and deploy many times so carry that same build artifact through push it to one environment test that environment push it to the next environment this pattern is really good for that so um if you enjoy the session there is you can tap up the top of the room to get more information and we'd really appreciate feedback as well hopefully this has been useful and thanks very much for your time everyone [Applause]

Channel: Amazon Web Services

Views: 4,101

Rating: 4.9402986 out of 5

Keywords: AWS, Amazon Web Services, Cloud, cloud computing, AWS Cloud, AWS Dev Day Australia 2018, CI/CD Pipeline, IAM, Identiy and Access Management, AWS CodeCommit, AWS CodePipeline

Id: HNSbnP673ls

Length: 30min 1sec (1801 seconds)

Published: Wed Aug 22 2018