AWS re:Invent 2016: Multi-Region Delivery Netflix Style (DEV311)

hello my name's Andy Glover I run delivery engineering for Netflix I'm really happy to be here in Vegas looking around I still see that most of you have not lost your shirts I basically lost my shirt Monday night and like every good person with a gambling problem I gambled on Tuesday night and almost lost my shirt again in fact got down to my last couple dollars and a good friend actually lent me some money to keep playing because I don't have a gambling problem and amazingly enough through his good advice I want all my money back plus his so I'm back to even and I'll probably gamble tonight so I love Vegas hopefully you're having a good time as well so again I run delivery engineer for Netflix and in this talk I'd like to share with you three things how we facilitate global continuous delivery the many listen lessons we've learned along the way and then consequently the best practices we now advocate so are there any Netflix members in the room thank you thank you we love you very much thank you for your continued binging you can now download videos to your favorite device hopefully some of you saw that it's good stuff so now you can watch all your favorite content on the flight home so thank you very much for for your continued support as you've probably figured out I work at Netflix I love working at Netflix it's the best job I've ever had and one of the the best things about working at Netflix and what I really really appreciate about the culture and the people on Netflix is that we are laser focused on winning moments of truth everything we do is focused all right first error of the there we go over clicking everything we do is focused on making your experience the best that can be whether it's to offer you know more content choices more supported devices or even better recommendations we focus so much on the experience because we want you to choose to Netflix whether it be on the plane or when you're working out or at your home we want you to watch it again and again and again consequently we strongly believe that speed is a competitive advantage we know that if we're not fast our competitors will be we also know that our customers you all demand that we're fast and finally the faster we learn the faster we can evolve to keep you happy all right it's pretty dry in here so earlier this year we launched in 130 countries and to facilitate a global audience we obviously rely heavily on AWS and accordingly we operate out of multiple AWS regions around the globe operating at this level of scale requires sophisticated tooling that ultimately makes it easy for development teams to move fast with a high degree of confidence and ladies and gentlemen that's why we built spinnaker spinnaker is evolutionary it's built on the shoulders of giants it is built on top or it takes the many lessons learned from multiple systems that have built been built throughout the years at Netflix that have facilitated continuous delivery since we went to the cloud over six years ago and spinnaker is a continuous delivery platform that orchestrates delivery of software assets whether those assets be an ami or a me a docker image a jar file a JavaScript application it doesn't matter it's flexible to handle what you define to be a software asset and an ultimately finisher facilitates moving fast at a global scale it accelerates our time to market spinnaker empowers teams to make rapid changes with confidence and thus enables a focus on building the right for you all as a platform spinnaker has improved developer productivity and efficiency and the good news is that spinnaker builds on the concept spearheaded by a previous cloud management platform called asgard asgard was open source many years ago maybe some of you and maybe some of you in this room are familiar with it a NASCAR did something really really spectacular for Netflix and hopefully for the industry and that is it it it created a model and in fact it created this concept of an application and an application is comprised of clusters and clusters are comprised of auto-scaling groups or aSG's everyone this room is an AWS fan so hopefully you all recognize an ASG is the top-level compute element in AWS and it's also tied to a region so this application centric modeling is key to maintaining a global infrastructure because you can have an app that's a global app and then has obviously aSG's distributed around the globe a key point I want to make about Netflix is that there aren't any centralized devops teams at Netflix for instance the spinnaker team we're focused on building and operating spinnaker not on building specific pipelines for teams as such teams at Netflix they own their own destiny and they make their own pipelines in spinnaker and so this is a key point there's no there's no release team at Netflix there's no centralized ops team that you know manages production if you're an app owner it's up to you to put that app in production you have total ownership over it so consequently if that you know if that app goes sideways at 4 a.m. on a Saturday night you're the one who gets the page and you're the one who's expected to solve that issue in a timely manner so spinnaker acts as a standard platform that enables all the benefits of continuous delivery at Netflix so it's it's really a paved it's a paved road teams are not required to use spinnaker they can go off the paved road but as the analogy serves it's kind of bumpy and you're kind of charting your own course if you're on the paved road you're able to move as fast as possible it's paved and it's a straight line between you know point A and B point eight maybe being some test environment and be being production and the good news is because it's a paved road it codifies all the best practices that that we as a company have learned and we can we can put them into spinnaker so that each team can take advantage of it and the beauty is if one particular team discovers something new or innovates something interesting they can then add that lesson learned or that component or that feature or that platform in the spinnaker and then everyone gets to take advantage of it and so as a paved path provides that standard means to plug into a greater infrastructure of integrated tools and platforms tools like chaos monkey which I suspect many of you have heard of it's it's incredible it's awesome and I'll get to it in more detail in a little while other platforms like our automated canary analysis platform which is spectacularly sophisticated awesome it's it's plugged into spinnaker so that everyone can take advantage of it and I'll talk about this more in a little while and there are other tools like this that are plugged into that paved road and that everyone gets to take advantage of it so Jays humble he wrote a book on continuous delivery in fact he wrote the book on continuous delivery he put it best when he said quote continues deliveries ultimately about making deployments boring and quote so what that means is like deployments are no longer this this event they're not this thing that you know back in the day it was a scheduled event on Friday all hands on deck we're deploying deployments just happen they happen all the time and in fact I Netflix we've been running spinnaker now for over two years and in that time we've seen deployments double from around 2,000 a day to four thousand today but I got to tell you in that time we have certainly seen our fair share of disasters and hopefully you haven't though due to the you know the amazing kind of resilient architecture with fall back that we have in Netflix a lot of the disasters that occur you're not aware of sometimes you might be aware of some of them but suffice to say disasters happen at Netflix and we've learned a ton of lessons that I'm here to share a lot of those lessons with you but something key that I want to stress and I'll stress it throughout this presentation is that we have a blameless postmortem culture and what's more is we feel very strongly that guardrails are far more effective than gates because we trust our employees to make good decisions we accept that occasionally failures will happen but putting draconian measures in place to prohibit the possibility of mistakes only serves to slow people down and probably diminishes opportunities for innovation so if you've ever gone to another presentation buy a Netflix employee or looked at any of the software that we've open-sourced especially if you look at spinnaker you've probably ascertained that we have some very strong opinions about how to do in this case multi-region software deployments that are both rapid and reliable and so I'm going to share with you some of these opinions I talked about spinnaker and and I'll show you examples of how we you know how we codify these best practices in spinnaker but I want to stress these best practices aren't unique to spinnaker they're their delivery best practices regardless of your platform of choice so first and foremost even though our particular tooling facilitates simultaneous regional deployments we don't recommend doing them in the event of a bad deployment for whatever reason that will eventually pop up be it a software bug infrastructure outage AWS hiccup which never happens keeping deployments focused to a region means your blast radius is limited as opposed to if you deploy to the entire globe then if there is a hiccup than meet your entire you know global audience is affected so what I want to do throughout this is show you examples of how we facilitate this within spinnaker so for instance in this case this is obviously a screenshot of fin occur if you haven't figured that out but this is a screenshot of spinnaker for those of you they didn't figure that out and this is a particular pipeline that is dedicated to a deployment in u.s. East where in multiple regions as I already said and what's more this particular pipeline also does something interesting that I'll talk about also shortly is that it leverages something what we call is a deployment window but the key point is and then if you look below if there's another pipeline that's deploying to another region now again spinnaker does not prohibit you from creating a pipeline that can deploy to multiple regions so for example if you deploy to two regions in spinnaker you can certainly have a single pipeline that either has to deployment stages or one deployment stage that simultaneously deploys the two regions we feel very strongly that that's a terrible idea to create one deployment stage that deploys simultaneously but you could certainly have a single pipeline that has two stages so pipeline is made up of stages and you could you could insert some sort of logic between those stages to perhaps wait verify that you know the previous deployment went well and then the next deployments okay it'll become apparent as I go through some of these more additional best practices why multiple pipelines are probably a superior idea but again our tooling does not prohibit you from doing what you deem necessary if you want to make one pipeline so remember in a fast-moving environment it's really tempting to push software globally all at once please take it from us don't do it take advantage of AWS regionality and limit your blast radius deploying to one region at a time deploy to one region at a time and ensure that that region is properly functioning before you go on to the next and if you want to you know think about it from a you know if spinnaker is a sailing term so think in terms of ships big ships unlike well yeah like the Titanic they're made up of compartments let's not think about Beckett Titanic let's think about a ship that actually didn't sink but ships have compartments right and so if the hull is breached you can seal up that compartment and ideally the ship doesn't sink same way to think about you know the globe and regions those regions are your compartment your bulkhead you can you can seal it up it only affects that particular region and the rest of the world isn't isn't affected so limit your blast radius please don't do simultaneous regional deployments so red-black deployments or as the rest of the world calls them Bluegreen this is V this style of deployment is a is your friend so a red-black deployment it takes advantage of AWS elasticity where the next version of a particular software package is stood up in a completely new separate ASG and once health checks pass in that ASG the previous ASG is then disabled then traffic is routed to this new one and so you have essentially twice as much capacity capacity being used but the benefit here right is that in this new ASG if something goes wrong with it you can quickly enable this one and then route traffic to it disabling this one and you've you know hopefully limited the amount of time that people are affected this is key this is this is a deployment strategy that's much different than let's say rolling deployments or in-place deployments so in this case the arrow is pointing to a an ASG that was deployed to EU u.s. one and it's enabled and you can see that they was red blacked into this environment because the previous ASG is in a disabled State and so you can see in this this particular stance if that ASG for whatever reason goes sideways maybe a bug surfaces later you name it something happens that team through either automation or manual manual steps which I'll show you next can disable that ASG enable that are enable the previous one disable that one and then route traffic to that thereby limiting how long people are affected and so in this case any US one that particular ASG is enabled there's a previous HD that has zero instance but nevertheless it's still around it's hanging around you know the and the launch config is still valid so in this case there's a button you can press enable rollback right and that'll rollback Ostend supply to whatever ASG you think is important or you would like to go to so red-black deployment as a as a style of deployment as opposed to rolling deployments offer the most rapid and reliable means to back out of a bad deployment you will have bad deployments eventually this is a superior means to get out of it so I kind of hinted at this at an earlier slide so Netflix is fairly our traffic is fairly cyclic our regional traffic is cyclic and for the most part people in any particular region watch more Netflix during the evening hours so for instance right now it's what is it like 1 1 20 the East Coast is beginning to ramp up people are getting home from work and naturally they're watching Netflix people on the west coast are at work so they're not watching West they're not watching Netflix hopefully actually I went to a company once and tried to check my eat my Netflix email and it was blocked any Netflix comm you know address was blocked in my browser it was kind of funny I guess they had a problem so yeah so our traffic is cyclic and so accordingly teams and Netflix can take advantage of the you know the peak in the trough so to speak and ostensibly they would want to deploy during a a trough and that's what deployment windows allow you to do so they they allow you to deploy during off hours so that if there is an issue it affects a smaller amount of people there's a side benefit to deployment windows which I'll get to in a second as well so in this case you can see that this is a configuration for a deployment and this particular team has decided that they will deploy to us east only from 10 a.m. to 2 p.m. Pacific and so back to the point about you know peak peak hours on the East Coast they're trying to hit a deployment before its peak and in fact we provide some more context in the form of a graph that shows a particular metric that we watch very closely which we call SPS which is streaming starts per second and that's again a very predictable number we we know again during the evening hours SPS is going to spike and during working hours it kind of goes into a trough so we provide this graph so we can provide that context the team to say hey look peak in the US a u.s. East you know is that that is at the top and so this team saying hey look we're gonna do a deployment before that but here's the other key benefit note the time is between 10 a.m. and 2 p.m. that's working hours so this team not only wants to hit this deployment before peak in US east but they're also they get the side benefit in that they have the most people on you know on the ground that can potentially fix any issues should they occur so deployment windows can limit the number of affected customers and the event of an issue but what's more deployment windows can ensure automation occurs during working hours when maximum coverage is available in the event of an issue so again I can't stress issues bugs bad deployments you know bad things will happen they oh it's you can't stop them and so in this case what we're trying to do is limit how long it takes to fix it so at Netflix as I also mentioned earlier we have a sophisticated telemetry platform that allows us to compare two different versions of running software we call this automated canary analysis or ACA and with our ACA platform we can compare two different versions of software taking production traffic and it's a great it's a great last line of defense to ensure things are working well before you open the floodgates and what I mean by comparing two different versions of software is is the metrics associated whether they be system metrics like memory CPU load disk i/o but even application metrics so imagine you you have version 10 of you know your service and you know obviously version 9 maybe is running out in production so what you can do is and up to aSG's version 9 version 10 allow a small amount of production traffic to go to him and then it's an apples and apples apples to apples comparison and you can you can set how long you want a CA to watch it and for example a CA may notice the the the newer version of your particular service has you know elevated error rates or a lower you know RPS or whatever it is it's up to you to decide what's important for your app and that platform will then tell you hey this is something's wrong something's gone sideways with your new deployment and this thing this tool this platform is highly integrated into spinnaker so in this case you can see here's a pipeline that's doing a production push and it includes an AC a step and in this case the AC a it was scored at 93 AC any scores are out of 100 and spinnaker allows you to define a go/no-go threshold so you can imagine maybe this team set a threshold as 90 and anything above 90 is is good to go anything below 90 is you know a thumbs down and the case of thumbs down then that particular stage would be marked as a failure in the overall pipeline would be in a failure State the cool thing is that with spinnaker then you can you can take some branching logic so to speak conditional logic based on that failure so you could say hey if this pipeline fails then run this other pipeline which would maybe roll back notify someone do something important it's really up to you to decide what that is but the key thing is even though this stage fails we don't necessarily leave it in a you know in a crazy state you can take cleanup correction you know correction corrective actions and part of the value of having this paved path is deep integration with other platforms so in this case you can easy you can you can get to a detailed report so you can click on the link and you'll be taken to the the a CA platform and it'll tell you why it scored this at a 95 you can look at all the details and you can see well what you know why did I lose you know five points for whatever reason this is important and I'll get to this a lot of these best practices are additive but note again this this deep integration and you can go to these other platforms from within spinnaker so the data combinatorics and a complex microservice architecture like what we have in Netflix it make makes it cost prohibitive to test every possible path before releasing something into production consequently ACA allows us to verify things are working well in production with live traffic and spinnaker makes it easy for us to back things out should you know this last particular stage or gate uncover some issues so invariably there are one-off infrastructure management tasks that people execute from time to time these could be you know emergency fixes or even occasional updates to infrastructure and it's easy to overlook automating these things but that's bad because manual tasks tend to create towers of knowledge by codifying this-this-this automation into pipelines anyone can run them with the benefit of consistency so here's an example of kind of one of these like you know once in a million events that thankfully this team automated here's a pipeline that can be used in an emergency situation to deploy to any region via parameter so you can imagine this probably happened once you know they they were running you know version foo and it was going fine and I'll send something when you know something went haywire and they you know probably manually figured out okay well roll forward we'll do XY and Z and thankfully this team then decided well we never want to do that again because that particular individual you know Cindy you know she likes to leave it you know 5:30 to go go home and this happened particularly you know that it happened to happen right around 5:30 so let's make a pipeline codify it and that way if it happens again at like 7:00 p.m. or you know whatever anyone can run this it doesn't rely on Cindy's you know knowledge here's another example of a pipeline that sets a runtime parameter or a runtime excuse me environmental property based on a parameter so we have an extensive dynamic configuration system at Netflix we call it persistent properties but think of it as you can have a running app and you can change properties of that app and that app can you know look for those properties so imagine a property like you know you know send or make house of cards live and rather than you know deploying a new app or restarting it for it to pick up that property you can go to a different console a different system and say house of cards live equals true and then any app that has that you know house of cards live parameter once it you know changes from false to true then can you know do whatever it means you need to do based on that parameter and that's done via this this persistent properties infrastructure but note that's kind of manual right you go to this other system and you find that parameter and you say true and all sudden you know boom the entire globe now has this this value is true and so with spinnaker what we're advocating is making that that's it's it's a deployment right it's a and you can codify that in a pipeline and so rather than going into this console and manually saying you know this is now true you can go into spinnaker and create a pipeline and say I want to set it to true and I want to limit the scope maybe I want it I want to set it to true but only for a couple instances over here once that's good to go then I'll set it you know to a larger scope in a larger scope so again that bulk heading strategy facilitated through pipelines within spinnaker avoids global disasters so you know automation is a no-brainer anytime a task is manual executed more than once consider it an opportunity to create a pipeline otherwise you risk creating towers of knowledge that become problems when those individuals aren't around to execute those tasks we have you've seen that some Netflix I'm sure every one of you has a story throughout your career where this happened and that's why it's a no-brainer daughter made this but you'd be surprised how many manual things every one of us do from day to day so email email is a terrible way to get someone's attention in fact I'm I'm sure in this talk alone the total email sent to all of us is well over a hundred thousand and twenty percent of those are from Nigeria and there's some other ones in there that I won't say ah so and Netflix we've seen on more than one occasion that emails requiring action have in a dirt and inadvertently been missed so therefore when it comes to notifying people of important events or actions that need to be taken such as deployments or manual stages we recommend using alternate channels like slack HipChat SMS anything but email so here's an example in this case here's a pipeline that uses email and slack to notify relevant people when this pipeline starts when it completes and most importantly if this if this particular pipeline fails the pipeline is specifically configured to notify a support channel in slack so this team really wants to be notified ASAP if something happens if something goes sideways and here is an example of a slack notification that a deployment into production completed successfully again this is important information that people need real-time or potentially want real-time status of that we are now aware that something went into production is helpful context should an alert get fired or something happens and we want to understand well was it related to you know what just what changed oh look you know this thing went into production it's probably related to that now here's a notification that requires a human to take action so in this case this particular notification is asking a human to approve a deployment into a particular environment so email was never intended to be a real-time communication mechanism consequently if particular events require immediate action to be taken use alternative channels like slack SMS HipChat you name it whatever whatever floats your boat but I would highly recommend you don't use email so who the machines thankfully haven't completely replaced all of us sometimes we're actually needed consequently spinnaker has a specific stage that we call manual judgment where someone I well a human must manually initiate a positive or negative acknowledgement so here's a manual judgment stage following an automated canary analysis and in this manual judgment stage there's an actual button for a human to press that either stops the pipeline or allows it to continue and as you can see following that manual judgment is a deployment into production so again here's this orchestrated workflow that facilitates a deployment into production with a kind of a safety valve right in the middle they you know the canary report could come out with something on a borderline and a human can look at it and say yeah you know what we've seen this before elevated you know whatever elevated exceptions they're transient it's not going to affect anything so we're going to go ahead with this deployment or vice versa we've seen this thing before even though a CA say it's alright we're I don't feel good about this we're going to stop this deployment right here and there so that you know what differentiates humans from machines is that we we understand nuance and frankly we have a gut and at Netflix we trust our engineers to use that gut manual judgment can be a powerful stage that makes sense in certain situations you all know what happens when you assume right like one person nod their head knowing your teachers never said this so uh uh yeah don't don't assume it makes a something out of you and me things change entropy is really a thing therefore a guard against it we have seen this time and time again but here's the epitome of recognizing that entropy is is something and in this is the epitome of not assuming reliability and it's chaos monkey if you don't know chaos monkey is a system that we run on Netflix that randomly kills running instances it's high it's tightly integrated into spinnaker in fact you get it on by default when you create an app you have to turn it off and so in this case it's it's part of the ethos of our culture at Netflix that demand service reliability we make no assumptions that if you know if your service gets shot in the head that it actually is gonna work so we tested all the time in production and in spinnaker you can configure kind of how you want chaos monkey to actually introduce chaos here's another great example of a pipeline that's triggered by a cron expression but note in this case this cron expression excludes the weekend this particular pipeline is deploying into production so you can tell if this team really wants to be sure someone is around in case again something goes wrong with this deployment and I suspect this this particular team also enjoys their weekends and we have seen this again as a lesson learned that you know automation that runs over the weekends it works a thousand times and then is that one weird scenario or doesn't work and it just happens to happen at like Saturday morning at 3:00 a.m. and getting a hold of someone at 3:00 a.m. is possible it just takes a little longer than you know looking over the person next you're saying something went wrong please fix this here's an example of a pipeline taking advantage of a stage we call a precondition stage within spinnaker so pipelines can take a long time to execute especially if you use a deployment window so imagine there's a pipeline that only deploys between 10 a.m. and 2 p.m. and you make a change to the code the pipeline's kicked off at you know 8 a.m. so that pipeline is gonna wait 2 hours before it froze that thing into production and the challenge you have there is that the underlying cloud you know that the infrastructure of AWS may have changed underneath you in that 2 hour span so we've seen this as well in that a pipeline is kicked off and and it waits for a little while and then someone else on your team manually goes in and changes something maybe they stand up a new ASG or they delete an ASG or they resize it it doesn't matter so the idea here is that again by not making assumptions by adding preconditions into your pipelines you can ensure things are as they should be before a particular stage executes Netflix believe strongly in the value of immutable infrastructure there are a number of benefits that one derives from immutable infrastructures first and foremost is consistency so what we do at Netflix is we bake everything upfront it takes a little longer to create an ami but then we use that same I am I threw environments and the ami has all its dependencies baked into it ie when we when we stand something up in production we we don't make calls to artifactory to download any dependencies or you know any third-party systems everything that app needs to run is already baked into that ami and so we have this notion of a find ami stage that you can pull an ami from any environment and pull it forward into a you know a forward environment so instead of creating an ami for every environment because there's a promotional stage here where you could go from test to pre staging - you know staging to production we do is bake once and use that same ami through all those different environments rather than baking each time and so here's another example of a find am i that's going to run before a canary and then that is going to go into production so the only change are the only constant is change and embrace it and construct pipelines accordingly let otherwise you run the risk that things will break because because that you base it on assumptions and I guarantee you when it breaks is the most inconvenient time the worst possible situation in production on a Saturday morning or Saturday evening when you can't get a hold of anyone in a timely manner so again these things are very much additive even if you follow every best practice that I've outlined here failure will eventually happen so you might as well make it as easy as possible to reduce the time to fix that that failure or that issue so in spinnaker we tightly integrate with Pedro GT so you can see here there's a button that if you know that there's a problem with this particular app you can go find the you know the little red Batman phone click it and it pages that team or that the on-call person for that team immediately so yeah we've made it super easy and in fact page of duty keys are required when you when you define an application we have learned some lessons from this you know our tight integration with Pedro Duty makes it super easy to link an application with a page of duty key but note we have actually we've learned that manually entering in Pedro Duty keys is error-prone I've seen those keys are like twenty characters just random characters and you know invariably someone's gonna fat-finger like the last character and then you know the whole paging paradigm is broken so what we've done is that we have we have created a linkage between Pedro duty service keys and apps and allow you to select the corresponding service named Titan integration with page or duty reduces the time to fix an issue by making it really really easy to get ahold of the right people so I said earlier we definitely advocate guardrails and accept that accident still happened as such we want to make it as easy as possible to get a hold of accountable people who can solve the issue as soon as possible so that you can continue to watch your favorite content on Netflix and so our tooling you know provides these guardrails to avoid major disasters but again I can't stress we accept that failures will eventually happen people make mistakes we're all human so as such we conduct blameless post-mortems so we can learn from these mistakes and not repeat them and in these post mortems there's no finger-pointing all of us have Netflix has probably done something that has broken something in production and we're we're still here to talk about it in fact many of the best practices I'm sharing with you you know that regarding how to effectively run in a multi region environment have come right out of these post-mortems so in summary if you if you really want to you know rapidly deliver software across multiple AWS regions please please please avoid simultaneous regional deployments use red black or blue grain deployments use deployment windows especially if you have cyclic traffic patterns or you want to ensure you have the most hands on deck during a particular deployment obviously leverage automated testing as a part your pipeline if you don't have automated canary analysis functionality at your particular company unit integration functional testing or obviously all a part of this mix they should be part of your pipeline automate everything especially there's one you know those one-off manual tasks avoiding email if you need to get a hold of people in real time recognize that sometimes humans they can make we can make good decisions sometimes better decisions that our computers please don't make a you-know-what out of you and me and finally make it really really easy to find accountable people so you can limit the time to actually provide a fix so Netflix is here at AWS reinvent because we're hiring if you want to work for spinnaker work on spinnaker or you want to do something else on Netflix please come by talk to me or any other Netflix are here a tree event we have a booth come chat with us we'll be happy to talk to you about all the awesome stuff we're doing at Netflix we also have plenty of stickers so please stop by and grab some stickers if you'd like to learn more about spinnaker we have a website spinnaker io there's a ton of documentation videos links into github repositories also there's a link into our slack channel and that slack channel is very very active not only is Netflix participating in it but other large corporations and people from the community it's a very very rich ecosystem of people that are more than happy to help you even better if you want to talk more about spinnaker if you have some ideas to share if you want to learn how we do things we'd love to hear how you do things come to booth 720 that's the Netflix booth and come chat with the spinnaker team will be there after this there's a six of us and in fact right after this talk we'll stand over here if you want to talk with us as well I will say I'm going to I left time to take questions so if you have any questions I can take them right now

Channel: Amazon Web Services

Views: 3,755

Keywords: AWS, Amazon Web Services, Cloud, cloud computing, AWS Cloud, AWS re:Invent 2016, aws reinvent, reinvent2016, aws, cloud, amazon web services, aws cloud, re:Invent, DevOps & Tools, DEV311, Advanced (300 level)

Id: 1HiilTXQo4w

Length: 41min 21sec (2481 seconds)

Published: Sat Dec 03 2016