HashiConf 2017 Opening Keynote

[Applause] [Music] all right thank you very much Seth we have a lot of really exciting things to talk about today I can't wait to share them with you but first what I'd like to get started with is just a brief talk about our history and growth to get to this point two years ago in 2015 we held our first Hacha comp ever in Portland Oregon we had over 300 attendees show up and it was at this house should come that we announced no man no man is actually a big part of this keynote today and we have a lot to talk about with nomad so you're gonna hear a lot more about that later then last year we had our conference in Napa California in 2016 and at this conference we announced new funding to help grow the business better cross-product integrations and as we started to commercialize for the first time ever we had customers come on stage to talk about how they're using hash work tools at enterprise scale and at that event we had over 500 attendees and today we're here in Austin Texas at Hashi comm 2017 and as Seth said yesterday we actually had almost 200 students and five sold-out trainings learning about Hofstra Corp tools and today you're surrounded by almost 800 over 800 actually total attendees here at hash accomp I see familiar faces I saw familiar faces in the crowd I saw some people that I know we're at Hacha comp 2015 and thank you so much for being so supportive of what we're doing and over the next two days we have over 35 talks plans and these are given by trainers speakers users kocchi kocchi Corp employees you're surrounded by the experts on Hashi Corp anything you want to find out you have the people around you to figure that out and so we hope you enjoy learning from each other as much as you do learn from the talks and so now I'd like to just talk about the progress we made in the past year about a year ago Hacha Corp was just about 50 employees and just over the last 12 months we've grown in fact over 130 employees and it's inspiring to see everyone wearing the Hoshi Corp logo continue to care so deeply about the community and our customers and in addition to this employee growth we've had really significant community growth as well last year we had our first ever hug events and this year we had over 42 hugg events with a total of over 7,000 members all around the world you could see a pin just about everywhere around the world this year we also had our first-ever Hashi day's events these were single track single track or single day conferences with about two to three hundred people and we did two this year we did one in New York and one in London and both of those were really really great events in addition to the employee growth and community growth this has helped fuel growth in the products themselves and so this year over the past 12 months we've actually released over 150 total new versions across our open source products over a 12-month period that is the most number of releases we've ever shipped these releases include thank you these releases include major features improvements as well as thousands of bug fixes we're continuing to improve our products regularly in addition to just the release numbers the download numbers are equally exciting over the past twelve months we've had over twenty-two million downloads across our product line and just in the past 30 days we've had over 1.5 million downloads which is the most we've ever had over a thirty day period so in addition to this amazing growth of our employees our community and the products we've also seen fantastic growth in our commercial products last year we announced terraform Enterprise and vault enterprise and earlier this year we released console enterprise through these products we've been able to grow revenue quarter-over-quarter and this allows us to reinvest back into our company and open-source communities and lets us story events just like this and hofstra corp wouldn't be anywhere without open source software and we and I absolutely love open source software but at the same time we're trying to ensure that we build a sustainable company that could continue to deliver high-quality open source software it's what we want and so what we do is we take features that assist with collaboration and operational needs of larger teams as well as governance and compliance concerns and we put those into enterprise products it's it's obviously not a simple decision we've been working over the past 18 months I'm building a framework so this is understandable repeatable and not no special cases and we intend to publish a blog post by the end of the year detailing that exact framework but thanks to this process and thanks to our open air our enterprise products we've been able to continue to grow and just on this slide you could see some of the logos of our growing customer base most of these logos became customers this year and I think it's really really amazing and ended it in addition to those logos these are the list of companies that are speaking at Hoshi Hoshi cough this year and presumably are therefore using Hoshi Corp tools which is absolutely amazing and from this growing customer base it's clear to see the trend that cloud adoption is in full force it's what our software does best every major company is either undergoing or planning a transition to the cloud and what this is looking like is going from a traditional data center a self-contained unit where if you zoom in you'll see a core infrastructure layer that's providing the physical resources necessary to run applications and an application layer maybe this is virtual majors virtual machines maybe it's schedulers that are that are deploying applications on top of this infrastructure layer and then surrounding that a layer of security and we're moving that into clouds whether it be private cloud public cloud Google Microsoft Amazon whoever maybe these companies are planning transitions to the cloud and we're here to help with that because while you do that a lot of these components remain the same you still require a core infrastructure layer you still need resources to run applications above that you still need an application runtime layer there's still a need for something to coordinate the deploys and operational tasks to actually run applications and then in this view of the world there's no more clear perimeter or network perimeter and so security just has to be a part of all of this you have to think about security all the way through and then with the cloud with multi cloud with multi data center being a default with all these different things there's a new challenge which is connecting discovering and communicating across these three layers and so what we try to do as a company what we try to do with our software is enable you to provision secure run and connect any infrastructure for any application and with that I'd like to begin talking about the product updates within these four categories and let's start with terraform so terraform over the past year has grown significantly the number of providers has grown by over 20 the number of resources has grown by hundreds and the number of contributors is well over 1,100 we've also established close relationships with all major cloud providers and are working together now to move towards a hundred percent coverage of all resources within terraform and this list is actually just some of the updates that we've shipped to open source just in the past twelve months and just as I said previously this is actually the most we've ever shipped into open source over that 12-month period and just to highlight a few we've shipped the ability to have conditional values within your configurations you have the terraform console to play around with terraform values in a rappelled type environment there's terraform imports so you could import single resources here and there that weren't under tariff or management before and then just recently we announced local values so that you could actually have local variables within your terraform configurations so from improved stability to rich rich new capabilities we've consistently been delivering to make terraform a better and better product and this is what's helping drive terraform to grow as much as it is in addition to the partnerships and community and all of this growth and maturity is really quickly solidifying terraform as the standard and infrastructure provisioning terraform could be used to provision any infrastructure for absolutely any application and in and striving to continue to make terraform this standard we're happy today to announce the terraform registry the registry is a place to find verified and community modules for reusable infrastructure components as well as a place to to get started and learn about terraform best practices we enter cloud providers cloud partners have seeded the registry with modules for the most popular services for the most popular clouds these include basic modules such as best practice network layouts as well as full application deploys such as console vaults and kubernetes and these work across all of these cloud providers we have verified modules which are recognizable by the blue checkmark and the verified modules mean that they've been vetted by Hoshi Corp both to work properly but also with the maintainer themselves to ensure that they're updating the module and keeping it compatible with the latest version of terraform and as well as the latest best practices of that cloud provider itself so for example Microsoft is actually writing the azure modules to ensure that you're working with Azure the way the Microsoft really believes you should work with Azure an ended ition - verified modules anyone can publish and consume their own reusable modules but instead of telling you how easy that is I'd like to show you [Music] and it's really that easy this video went from having no account and never using the module registry to signing up publishing a module and having it available and it wasn't sped up at all you could go from nothing to publishing a module in about 15 seconds and as you can see we surface a lot of information about modules and one of the most exciting things is the module registry brings the terraform the concept of module versioning every module within the registry has a semantic version associated with it and we allow you to navigate those versions within the registry you I additionally when you when you publish a module with the registry what the registry actually does is bring in the module parse the terraform code learn about it and use that to generate help and documentation about that module so you could see on this page that we have the readme rendered we have a list of inputs a list of outputs the list of resources it might create we also parse out a list of other module dependencies whether they're in a module registry or actually anywhere else get s3 anywhere else and we show that to the user and you get that automatically there's no metadata you have to type up we parse that out of the terraform code itself and the terraform registry is available today right now at registry terraformed IO today we've also released terraform version 0 10.5 which allows you to use the module registry modules and later this year before the end of the year we'll release terraform 0.11 which contains significant improvements to the module system including the ability to constrain by module versions but every module that's in the registry today is already versioned and with that what I'd like to do is invite Corey Sanders from Microsoft on stage to talk about terraform and the module registry good morning everybody excellent Wow a lot of people here this is great this is my first time actually at Hoshi conf is well so I was raising my hand backstage and one of the biggest things for me was trying to figure out what outfit I was going to wear this takes a lot of deliberation when I'm up on stage probably could be spending more time working on products but instead I'm worried about what I'm wearing and I wanted to make sure I represent as many of people in the audience as I could and so I wanted to make sure I got you know the jacket for enterprises in the room so please focus on this part of my outfit okay and then for for startups feast your eyes on this so now we got a little bit of the a little bit of Star Wars going on underneath so and let's do it again Enterprise startups good okay great all right now everyone's ready all right so one of the most exciting things about working with with Hoshi Corp is has been work around terraform and it's something that we came out and announced actually just a month ago that we were going to be increasing a lot of investment to make this work really fantastic on Azure and I'm so excited just a month later to be here on stage and to be able to walk you guys through some of these things that we've already put in place and so first as Misha was just talking about the terraform module registry this thing is fantastic announced today we actually have released eight certified as your modules that you can use starting today and it just makes it so much easier to do complex things in a really really easy way so you can go check this out we've got support for compute we've got support for networking we've got support for sequel and of course consul involved also integrated as well I mean so all this is available right now and to give you a sense of sort of how easy this is and how simple it creates makes it - for you to create resources if you take a little look here at this example if I were just creating a network on Azure today using terraform you can see I have to go in I have to create my virtual network I have to create my subnets and define what those look like I have to create the security groups I have to create all the rules and this is great if you want this type of control you want this type of configuration option you can go do it but what's great about using a prebuilt module is it's incredibly easy for just getting started in many of these scenarios and so this is what it looks like using our module for the network and so you can see very very simple just a few lines here to be able to create that same full network that I showed you and just with when you go through initialization it pulls down the module and away you go and so this is how easy and how powerful this new capability is with terraform and I'm really excited to be launching today with eight of those certified capabilities built right on Azure and so that's fantastic one of the other aspects of the partnership that's been really fun is one of the more beloved features of Azure is our asher cloud shell so the azure cloud shell is built into our portal experience so if you go to our portal experience you can open up our shell it's effectively a command line in the browser and behind the scenes we're spinning up a Linux container on your behalf so you can run commands and it's got a bunch of commands pre-installed and pre-configured and one of the things we are announcing today is as part of this command line experience we have terraform pre-installed and pre-configured for you to be able to run commands against this and so let me let me actually show you what this looks like so this is the command line experience right here in the portal and you can see terraform is available there and if I expand that out you can see just how easy it is to get started and so it's already been configured it's already been set up with credentials and away I go with initialization and I can deploy and just how simple this is there's no no additional setup there's nothing else to really learn you can get going with terraform right on azure almost immediately and the beautiful thing about this this goes anywhere you go so anywhere you go you can open up this portal experience at any browser any any type of uh oh s a Windows Linux a Mac you can run this and get it going the other aspect that you can actually run it on your phone as well so iPhone Android you can pull this up and you can run these commands and by the way of course this isn't restricted to only run against a sure you can you to run commands against any cloud if you so choose and so this gives you really a nice experience to be able to get going with terraform or continue going with terraform if you're already there so this is exciting like I said we're announcing this today so go play with this and it is available directly in our portal experience and then the last thing to talk about is our provider momentum this is something as part of the announcement that we made about about a month ago we made the promise that we would make our provider updates we make sure we kept up with any of the announcements that we're making and and kept the providers to date we have launched two new things in the last month and both are already available in the azure provider for terraform so these are within days these are available for you to be able to use directly with terraform and so examples of this event grid which is a events as a first class citizen in Azure so event objects available available today and our container instances this allows you to spin up container instances just like infrastructure within seconds and they're billed per second and this is also available today so it's been fantastic the partnerships been wonderful we've seen a lot of exciting customer usage it's been great to work both with the community and with hashey Corp on all of these great investments and deliverables and really looking forward to the partnership has really just begun so there's so much more for us to do if you've got feedback comments I'll take them right now no just kidding come to our booth and and meet up with us we'd love to talk with you more about the work we're doing and with that come on back up Mitchell all right thank you very much Cory it's all super super exciting stuff and so the next thing I want to talk about is how provisioning infrastructure is of course not a solo activity it's a collaborative activity multiple people work together to build and maintain infrastructure and over the past year we focused heavily on in creating these collaboration features and an open-source we shipped remote backends which are a marked improvement over a remote state we've also introduced state locking whether you're using s3 or console or other backends directly into open source to try to make collaboration safer and more easy but we also have terraform Enterprise which is our tool that's purpose-built for safely collaborating on infrastructure and over the past year we've spent most of that year redesigning and re-architecting terraform Enterprise to feel like a much more natural extension to terraform a user of terraform should be able to adopt terraform enterprise enterprise and feel right at home with a lot more power and a lot more safety and so this REO architecture was done in collaboration with some of our most loyal customers and we're excited to show you some of it today the first thing I want to show you is workspaces we've made workspaces the primary place of work within both terraform and terraform enterprise so when you're adopting just terraform on its own you have the terraform workspaces command you don't need to use stare form Enterprise to get that workspaces our core part of how you work with teams with terraform using any kind of back-end but when you use terraform Enterprise we list your workspaces right here in an easily browsable away with the workspaces you could view each one you could view its history and you could view who has access to it as well as what they're currently doing and when you adopt terraform Enterprise terraform actually offers to migrate all your workspaces from whatever they're back and they're currently in over to terraform enterprise and equally if you choose you don't want to use sera from enterprise we offer to migrate them back out and so adopting terraform Enterprise using and using Tara form with Tara from enterprise is now a much much more seamless experience we've also redesigned the plan and apply pages to look a lot more like and also just surface the information you want to see and so when you click into a workspace and click into a run this is actually what you see today you see the run who triggered it the commits who that triggered it that links directly to bitbucket or github or gitlab whatever you're using you see the plan who confirmed the plan who said it was okay maybe some comments there I'm not sure if we could see it but you can see the person reviewed the plan if you click view plan you could actually see what was planned and then it applies all in one place and this is all stored in the history is all visible and we've also rebuilt air for an enterprise on top of an API first architecture so it's built on top of an open API specification which lets you which lets you control every aspect of terraform Enterprise via an API so now you could build automation around terraform Enterprise integrate it into your existing software and as well as just fully configure it using infrastructure as code and you stare from enterprise and then finally we've moved the home of terraform enterprise to App terraformed IO to better represent that it's simply an extension of terraform and something that improves terraform with teams terraform Enterprise is open this new terraform enterprise is available as a beta today and will open general signups before the end of the year and next what I'd like to do is invite our maandag our hashtag or founder and co CTO to talk about some more exciting product updates [Music] [Applause] thank you so much Mitchell and thank all of you guys for being here we really could not do this without all of you being a member of our community and showing up and supporting us so thank you so much you are what makes all of this possible so I'm super excited to be able to start today by talking about vault we open sourced it about two and a half years ago and few people know that it started life as an internal project it really started for us looking at how we solve our own secret management problem and in sort of talking with users talking with customers about the challenges they're facing they really convinced us that this is software that we should open-source there's actually this much bigger problem with the way we think about and handle modern sort of infrastructure security and that bullet can play a pivotal role he'll role here and so over the last two and a half years what we've seen is an incredibly broad adoption of all honestly much much more so than we even expected ourself when we opened it and I think the lesson for us is that modern security software is becoming much more important than ever before and so the question for us when we started trying to understand this was what's really changing why is this happening now that all of a sudden there's this sort of renewed focus and and sort of excitement around things like vault and I think it really goes back to what's changing about how we think about network security how we think about application security when we kind of look at the way we used to do it it used to be sort of this castle and moat approach we had these fortified walls that were the outside of our datacenter we had a single drawbridge which was our ingress and egress and we layered our firewall and our intrusion detection system and our Web Application Firewall and all of this sort of network based middleware that provided the security and then we said once you're inside that castle everything is safe right we don't have to worry about it because we secured everything at the entry and exit well we've seen over the last few years is as you go multi data center as you go public cloud as you go public/private if these network topologies get more and more complicated and so you go from a single drawbridge to two drawbridges to ten drop bridges and at some point you really have no idea how many draw bridges there are anymore because you have mobile devices that are on corporate VPN that's linked back to the sort of prod fabric and so it's almost impossible to even understand what the network topology looks like and so this starts changing that assumption where before we said you know what we can just say if you're in our network everything is secure we don't have to worry about it so now we have to rethink this a little bit we have to actually consider that you know really we have to protect the infrastructure even if you're on our network our applications should not assume that you're authorized that you can do things just because you're on the network right we've learned that this is sort of a fatal assumption and we can see these fatal assumptions sort of in the news all the time and so with vall what we really think about is okay how do we start solving this how do we change the assumption of security and then what do we need to do to address it and there's really three totally different axes that we see the first one which is really where volts started life was secret management was looking at things like database credentials API keys TLS certificates how do we securely distribute these things to our application if we're no longer hard-coding it in the app or storing it in plain text we really need to have sort of a more sophisticated automation around how we manage and distribute those things and so that's really what we mean when we talk about secret management is how do we get these sensitive credentials to our applications that need them in the data center then when we started thinking about this problem and really building out the secret management side of things it became clear that one type of secret a very special type of secret was encryption keys right on the surface it's not that different from any other type of secret it's just a short string with some bits but then what became clear is if we treated it like a first class concept and we sort of thought about it in a different way then all of a sudden we could provide these much higher level api's we could think about keys as things that we're versioning we can provide high level encryption api's decryption H Mac all sorts of sort of cryptographic operations and so it goes from just thinking about it's a string that we have to hold to it's the sort of first-class encryption as a service tool so that we don't have to worry about implementing cryptography correctly in our end applications instead we can just call out to vault which is doing it for us and so once we got this far along you know then the question became well why do we have this two different sources of truth one system for human operators one system for applications how do we keep things in sync this is just sort of you know both bad practice from a security standpoint and introduces a possibility that we're not going to do things right so how do we centralize everything and so this became the next axis for us which was how do we do privileged access management and make sure that the system is easy to use for human operators who need to interface with it so that is a lot of things to tackle three totally different kinds of subproblems and so over the last year we've been incredibly busy on the in the vault team adding all sorts of new features and capabilities to better do all three of those things right and so we don't have time to go over all of these things you know I think one of the ones that that was most exciting over the last year was the ability to have secure plugins so that we can really continue down that mission of there should be a single way to broker access to these secrets regardless of the backend technology so you know while we can add support and we are adding support all the time for new clouds and new databases and new systems the plugins let us really extend that last mile to just about everything and now I'd like to welcome Dan McTeer on the stage from Adobe to talk about how they're using ball in their environment [Applause] [Music] [Applause] [Music] everybody down here glad to be here every time I'm in a situation like this I'm reminded of a poor life choice I made in college we had to take public speaking it was a required class which i think is a great idea the semester that I signed up for it they happen to be offering it online so in spite of understanding the irony behind that I went ahead and signed up anyway so I'm gonna dig really deep into all those experiences I've never had and used those to my advantage today so I worked for Adobe Digital Marketing for those who aren't familiar with that we actually process over 100 trillion transactions every year for our customers more than two-thirds of Fortune 50 companies are our customers and that includes a to the top 10 internet retailers all of the top 10 commercial banks all the major automotive industry manufacturers so we have a pretty big customer base pretty critical things going on our customers exist in just about every market and every industry you can imagine in 2016 we provided a report that predicted the the online sales for Thanksgiving and Black Friday and Cyber Monday within 99% accuracy so we have we have our hands and a lot of a lot of different stuff going on online if you've ever been on to like a major TV premiere cable provider channel like HBO or something like that and ask for your cable provider that's one of our products if you've ever streamed the Super Bowl on a mobile device also one of our products so our product suite is made up of more than 23 different digital marketing tools and those are built and maintained and operated by over 50 different teams developers operations staff and those teams we like to embrace diversity right at Adobe we do a really great job of that but part of that includes everybody having their own opinion about what tool where it's best and and using all of those tools to manage our day-to-day work so those products run on a foundation of more than a hundred thousand hosts all over the world in four major regions and they span across to Azure AWS and Adobe data centers my team is part of a larger organization that provides core central shared services for all of digital marketing that includes things like DNS get DHCP my team in particular provides security related services so things like LDAP Splunk vault obviously so the opportunity for sprawl in situations like this is just off the charts there are so many different tools to use in the different clouds different open-source tools available you can have one piece of information over here and a completely different piece of information over here how do you keep those things in seek and sink and on top of that we are constantly fighting the Battle of you know people have so much time to do certain things security requirements compliance requirements across all these different industries that we have to abide by make it difficult for our operational staff to to kind of keep up sometimes so our team philosophy is that we take on the brunt of the security and compliance requirement work but we make sure our services are easy to consume are flexible so that other teams can spend a lot of the time focusing on the customer on the operational side so we're always looking for tools that like I said are flexible are easy to use have a REST API REST API is key because everybody has different configuration management tools everybody has processes they follow we want to make sure that those things are also simple to deploy so any market we pop up in we can be ready to go in in a matter of hours with our with the services that we provide so the choice was simple from our perspective we've had tools that our industry leaders in secret management deployed at different parts of Adobe vault Enterprise was the clear decision for a lot of different reasons it's great to see the message today that's how agnostic these tools are how many different places they fit into and and vault is obviously no exception to that so we have clusters deployed in four major regions across the globe each of those regions has three at least three vault clusters one in AWS one in Azure and one in an Adobe datacenter and those are all linked together by an underlying network fabric if you have questions about that later on please feel free to talk to me this has done a number of things for us that have just been fantastic a couple years ago when I started working at Adobe we had a secret a privileged account password rotation process that would take sometimes two to three hours every time we would rotate the password with the implementation of vault across the globe and and some updates to those processes we've cut that down to about 30 seconds every time we rotate those passwords and when you're talking about thousands of hosts tens of thousands of hosts that's a huge deal not only that we've integrated that with our our Bastion host solution so that you can do privilege escalation again this is all super easy to do because of the flexible REST API and then whether our users are accessing from Southeast Asia and Azure or Us East one and AWS or an adobe datacenter everything will be the same it's all standard there's no having to guess what's gonna happen in any given region based on on what you're pulling right so so standardization has been key replication in Vault Enterprise has being key to our deployment and then of course our auditors and our security people are super giddy about the information that gets provided in the audit logs there there are so many positive things that we've seen just in the four or five months that we've had vault enterprise deployed to production my marriage is improved my kids respect me now thank you it's a fantastic product I can't say enough good things about it later on today 5:20 I believe Chandler Alphin are our primary vault deployment engineer and manager of that service will be speaking on track' so please you know come by and see his presentation come talk to us ask his questions we'd love to answer questions thank you for the opportunity to be here and all welcome Marmont back on the stage [Applause] [Music] can we just give dan one more quick round of applause thank you so much Dan for coming and sharing that with us so earlier this year we started working with our friends at Google many of whom are actually here so thank you to the Google folks for being here and supporting us but we started engaging with them earlier this year on sort of advancing Volt's core mission which is how do we support the users wherever they want to be our goal is really to look at providing a secure workflow to secure your infrastructure and your applications without being prescriptive about where you're running so we're always listening to users and community members to understand where do you want well to run where is it not working well for you and something we heard over and over was please add support for the GCP platform so we started working with our friends at Google and they've helped us to implement the Google I am off back-end which we launched earlier this year and that's been awesome so today I'm super excited that we get to talk about working with those same group of folks to add a native support for kubernetes probably right behind everybody asking us for GCP support has been everybody asking us for kubernetes support for vault and so this is something that we've spent some time prototyping and spiking and working with the community to understand how to best do this and we're really excited that you know we feel the implementation we got to is sort of the simplest and best of all worlds so just to talk really briefly about how it actually works now when you launch an application with kubernetes it's able to get a native kubernetes service JWT token with the kubernetes auth back-end we're able to take that token and provide it to vault vault will then locally do a cryptographic verification to make sure this token matches the role this token cut is originating from the kubernetes cluster and that we have a strong guarantee that you know this service is who it claims then we go even further than that and call back into the kubernetes api once we've done that check to verify this token hasn't been revoked it hasn't expired it hasn't been deleted for some reason and once all this is done then volt will generate an auth token and provide it back to the service to scope down to exactly that service roll in configuration and at this point the surface has a native vault token so it's able to directly communicate with volts API make use of everything from static secret management dynamic database passwords encryption offload it's not intermediated the app can do anything that bolt really supports and so what we really really like about this integration is it removes a bunch of intermediate moving pieces you don't have to worry about anything in between the kubernetes api and vault there's nothing between that service and vault everything is direct and there's a native integration and so we're super excited about how that ended up working so this is landing in vault 0.83 that's gonna be available today or later today so please download it check it out and I think one of the important takeaways is that you know this is part of our ongoing effort to work more closely with the kubernetes community earlier this year if you're a terraform user hopefully you saw we announced the native kubernetes provider which has support for the most common resources and we're continuing to expand support for that and so this effort is part of that as well as we try and expand our support for kubernetes across the portfolio so I think you're gonna see a lot more from us in that and we'll talk about that as it shows up so now I want to take some time to talk about console console is our service discovery tool and the challenge in talking about console in some sense is that it does so many different things right so when we talk about it being a service discovery tool what does that really mean right in some sense our goal is to provide a toolkit of features that really solve all of the different aspects that you really inherit the moment you go to a service-oriented or micro service architecture the moment we go from one to many machines now we have a service discovery problem how do we actually know where our upstream services are running we have a health checking problem because now that there's many up streams some of them may not actually be healthy we have a configuration problem how do we configure our many different services we have a coordination problem if we want to run multiple you know instances of a service for availability and one of them fails and we want another one to take over we need distributed locking tied into health checking tied into discovery so the list of kind of things that console needs to support to really effectively able to solve this service discovery problem is broad and so what you'll end up saying then is a huge set of features and capabilities that we're constantly adding to support this so this is just a small preview of all the things that if console has gained in just the last year and one of the things that's always been super important to us when we thought about console when we first built console when we designed the system was realizing that because of the role it plays because you're depending on it to do this service discovery this health checking this configuration it acts as kind of a dial tone to the data center all of your services are using it to look up their upstreams health checking is being used to do automated fail overs and so it's very very sensitive and if it takes down it can have a lot of collateral damage to your data center and so from the very beginning it was important to us that the system be rooted in academic research and that we're really delivering a world-class state-of-the-art system to make sure we're giving you the highest level sort of availability and performance and ability to tolerate failures and so it's really two very important sub systems one is consoles gossip subsystem which allows it to scale linearly to data centers with thousands of nodes with an effectively be idle and this gossip subsystem is based on a work called swim the other really important subsystem is its consensus and replication system this is based on raft which is simplified paxos which is a consensus algorithm and so what this lets us do is replicate console state between multiple servers and gracefully handle the failure of any of those servers and so this lets us give you these guarantees that the system will both scale and stay online so in the last year we were really excited when we announced the launch of Hasek or research and our goal was to really recognize that we consume a lot of this research from academia and use it to build our products it's at the heart of many of the tools we build and so we wanted an opportunity to give back as well we wanted to be able to do our own novel research into algorithms into system designs into empirical analysis of what we're doing with these things and then publish that and give back to the community that we owe so much to and so I'm super excited that in the last year we published our first-ever paper and we call it life guard swimming with situational awareness you know you might claim we got a little overly clever with it but you know we had fun at the title the core sort of inspiration of this was really actually working with one of our largest users who was experiencing a denial of service attack so they were under a denial of service and their front end machines were banning just totally slammed and then they noticed that for whatever reason random machines that aren't on the front end consoles declaring as failed right machines that are not publicly accessible to the internet that are not affected by the DDoS are just being marked as offline but are being marked as failed and so this was a really interesting opportunity for us and so they brought us in and worked super closely with us and gave us access to their environment to really understand what's going on under the sort of pathological condition of the ddos and what we found is there was a number of shortcomings in the underlying gossip protocol that it made certain assumptions that under cases like this where some nodes were heavily degraded would degrade the entire clusters performance so we work with them and we worked internally with Hoshi Corp research to come up with a number of novel extensions to the waste swim actually operates to work around these issues and the results are kind of amazing honestly they beat even our own expectation what you can see on that graph is basically sort of the baseline system on the left which is pre lifeguard and just by enabling lifeguard no other changes no other configuration for the system we can see a 98% reduction and the rate of false positive in the system and so this was kind of crazy for us because some of our customers run data signs was tens of thousands of nodes hundreds of thousands of console agents in a single cluster and so this ninety-eight percent reduction for them is huge console basically goes completely quiet in terms of flapping and so we were able to take some of this benefit some of the sort of 98 percent reduction and actually tune the system to not only have a dramatically lower rate of failure detection but also to detect failures even faster because there's so few false positives so we're super excited about this our director of research john curry is here he's going to be giving a talk about life guard so if you're interested in the nitty-gritty of how this actually works please make sure to attend his talk one of the things that sort of come from this relentless focus on operational stability and scaling has been that we've seen an incredible amount of adoption for console right it is now deployed in some of the world's largest video streaming companies some of the world's largest financial institutions some of the most trusted news sources and some of the largest consumer tech companies and partly it's because of this focus is the sort of relentless polish we put into the system that's that it actually works at their scale and this has not been overnight by any means Council has had a long long history you know many of our earliest users have suffered through some really atrocious bugs and we apologize and we've come a long way since then and so this has been a slow and steady but relentless march to improving the system I think many of you probably know where I'm going with this and so we're very very excited today to finally announce console 1.0 it was really only you know now it's our third product vagrant actually just had two 2.0 in the last two three weeks here but it just joins vagrant and Packer as Hacha corpse only 1.0 two products and for those of you who've never had a chance to sort of read up on our philosophy of it it's not because our tools aren't used in production it's not because they don't see scale it's just that hot record holds such an incredibly high bar for what we're willing to call it 1.0 we really want to make sure we understand what is the total surface area of the use cases how are people using this thing does it work even at these most massive scales and make sure that it reaches our level of fit finish and polish before we put that 1.0 label on it so rest assured this is not an end-of-life announcement for console there is a you know long roadmap ahead of it you know sometimes it just takes us four or five years to actually get there so console 1.0 is going beta today it's available download it check it out let us know if there's any last-minute issues before we take at 1.0 gold master next I want to spend a little bit of time talking about nomad Nomad for those of you who've never played with it don't know about it is our batch and service scheduler and so it's goal is really to enable you to specify whatever your workload is whether it's a cron job that you want to run every our weather it's a you know spark job that it's doing big data processing or there's a long-running service and just specify at a very high level what you want Nomad to do and then no Matt to abstract the details of how to run out where to run it all of those details as you might imagine for a system that's trying to do all of these things there's an incredible number of asks there's so many things people want the system to do I want rolling deploys I want blue green I want Canaries I want manual promotion I let data dog Prometheus so on and so forth the sort of surface area of what people want because this thing sits so core to the data center is broad and so we've been super busy adding all of that so Nomad this is just a small set of the things it has gained in the last year it's been sort of an incredible March to roll out all of these features and so I'm super super excited to welcome Mohit Arora on stage to talk about how gedcom is using oh man my name is Matt my name is Matt and I'm part of cloud platform team at jet for those of you who are not familiar about jet jet is an e-commerce company based out of New Jersey we are part of Walmart family now jet was acquired by Walmart late last year our value proposition for our customers at jet is that prices drop as you shop so as you build bigger baskets as you add more items to your shopping cart our system in real time tries to compute the savings based on the supply chain efficiencies that can be brought in to fulfill your order and we pass on that saving back to customer - a quick example if you are buying a shortage at $25 if you are buying a second shirt at jet normally on all other e-commerce platform that will be $25 but here if it is coming from the same merchants can be shipped in the same box these are the supply chain efficiencies we are talking about and the second shirt that you buy a jet might be twenty dollars because of the supply chain efficiencies that can be computed in real time so jet was born in cloud in 2014 and we all the systems that power gedcom run in public cloud which is Microsoft Azure in our case our journey with cluster scheduler started late last year when we were trying to bring the bring the utilization of other platform up we are trying to drive the cost down so we were trying to compare all the schedulers which are out there in the market and we ended up comparing nomads and some other schedulers which are available so apart all these distribute schedulers promise you similar things but no mid one for us primarily because of these four things so first is cross-platform so we we our applications are built in F sharp and our cloud platform tools are written in golang so our F sharp services run in Windows in production so we wanted to have a distribute scheduler which can support both Windows and Linux and nomad was the clear winner for us the second is flexibility so even though we were big on containers but we had legitimate use cases where we want to run some workload out from as a general-purpose workload on the instances not packaged as a container and nomads also gives us that flexibility that we can either run a general-purpose workload or we can run a container ease of use so we were heavily invested before Nomad in console and gotten nomads and console have same semantics so we already knew how to run console and production when we tried nomad out it was pretty easy to get started with so that also played a factor in our decision the last is the fantastic console and vault integration as I said we were we had a long list of use cases that we drive out of console and Nomad has out-of-the-box integration with both console and vault and that was also a factor when we were making a decision so after we picked Nomad we built an ecosystem around it so we we never exposed Nomad directly to developers so what we essentially ended up doing is that we built a tool in front of Nomad and all we asked developers to do is in their version control system they check in a file called manifest file a couple of manifest files one is service manifest but essentially a service manifested watch your services what our health check of its service of that service and then the deployment manifest file deployment manifest talks about how do you want to deploy that service if it needs to be co-located with other services and all that and rest of the magic is done by this tool that we have built which we call gizmo in front of Nomad gizmo does all the stuff and then ultimately it creates a nomad file that it deploys on the nomad cluster so when we kind of took this Nomad as a platform and on top of that this tooling we exposed it to developers these were the immediate benefits that we saw the first is continuous delivery story developers love the fact that they can define how their services needs to be deployed in version control system and then rest of the stuff is taken care for them by gizmo but they can change go back to the version control system and done the next run of the build and it will update the configuration for them the second important win for the the app dev team was G redundancy story so in in Jett we run services in multiple modes some of them we call hot hot like depending on what it is doing or hot warm hot cold so we if you are writing a service it has to opt for one of these models which depend which further defines how it will be deployed on on multiple regions that we run out of so with this journey in the manifest file developers can pretty much define which mode they want to run their service in and everything else is taken care for them and they love this fact scaling so before Nomad if if the app dev teams are worrying about scaling they are worrying about the compute infrastructure as well so in this offering what we have for them is that we have a separate process which keeps monitoring the Nomad infrastructure and if it needs to be scaled up it scales that up and developers pretty much have the guarantee that if they need to scale their service up it will be pretty quick scaling effort for them because the infrastructure is already scaled up for them they don't have to worry about it the next is operational so in the service manifest file as I talked about developers can define health checks we also let them define the page duty service key and on on all the ecosystem that we have built we keep monitoring their services on their behalf we have a handle on their page of duty key if it is crashing and then we we can page them so they love this fact that they don't have to do anything everything gets taken care on the eco system that's built on top of nomad the last is the canary deployment and experiments whichever business users love so as like any other e-commerce company we run a lot of experiments at jet and we kind of try to figure out what's the conversion related to each experiment so that we can run that experiment longer so no matter lao's us to run multiple versions of the app on the platform and then we have built a tool on top of it which we call phaser which allows you to face the traffic from one version to another and then we do some things at the proxy layers at the headers and business users can correlate what's the conversion impact for each of the version that's running in production and huge when again so this is how our production cluster looks like I know it's not a huge cluster in any way but where we are right now is that we have proven the platform and we have proven the echo system that we have built around it and it's a completely self-service model and at this point of time there's a greater push within the company to move all these services which are not data stores on top of nomads and before holidays the idea is to run everything that we accept data stores on top of nomads and in early November I expect this number to be going up close to 400 nodes and as we speak 50% we have already migrated the front end which is a node app for gedcom on nomads and we have faced 50% of our traffic on that version of the application and before holidays we plan to go 100% and all other services even though they are not serving customers directly will also be running on top of nomads so yeah that's it do check us out during holidays we'll have some great deals going on as well and the platform and the platform will also be running on oh thank you can we fit one more quick round of applause for Mohit sharing gedcom story so something we've seen over the last year is increasing adoption bite for nomads both at you know small organizations and large ones like jet comm and a frequent piece of feedback we've got and is it can be really hard to onboard new users onto nomads in these organizations you have a core that has operational sort of expertise and knows how these systems work and understand demand and depth but for everyone else it's sort of a new mental model to learn it's a sort of foreign when they start trying to learn about the operational side of things and today really Nomad exposes everything through an API and a CLI which can be sort of unfriendly for some of the users so to help make that a little bit better we're excited to have a first-class native integrated UI that we want to show you today so this will operate very very similar to the console UI if you've ever used that it'll be integrated into the binary you just flip a single flag and it'll be served from the same set of processes and now you've got a native experience that we're committed to adding support for all of nomads features here's just I'm just gonna show through a few of the screenshots of it it's a much richer interface I recommend downloading and playing with it but here it gives you a feeling for what it's like when you have multiple jobs running on the system and at a glance you can see what type of job it is the priority what state the various allocations of the system are in on the other side is what we get if we drill into a particular job if we want to understand what's going on with this particular application and here it's sort of highlighting one of the features of that we ship earlier this year which is nomads ability to canary deployments so in this case the application is sort of canary in stencil to run the new version and it's waiting for an operator to come in and do a manual promotion so it's deployed out the new canary it's running and it's just saying whenever you're ready give me the go and we'll continue the rollout of the rest of the service and so this an example of the kind of things that we want to make more visually obvious to users make it easier for developers who are self servicing and doing canari deploys to be able to do that on their own here's an example of another feature we ship earlier this year which is multiple job versions so now nomads will track the last few versions of a job and one of the reasons we're doing that is that we can show you what's changed so if there's an anomaly in the performance the service isn't acting exactly as you expect or maybe it's acting better than you expect and you want to know what changed why is this happening you can go into nomads interface and ask it to basically show you the Delta what's change between these two job definitions it might be something as simple as just the version of the job it might be that the resource allocated to it is different but now you can sort of drill in and see these versions and if something's wrong and you don't like the current version you can push a button and revert back to a previous definition of the job on the other side you see what it looks like when multiple jobs are running on a single node so this is sort of a node oriented view of the cluster as opposed to a job oriented view so what Nomad is doing in the background to increase resource utilization to solve the problem that folks like gedcom are saying is it's trying to BIM pack multiple applications onto the same node typically what we see in most clusters is less than five percent utilization and a world without schedulers and most of these resources are just idle and so what nomads trying to do is understand what are all the resources available and try and fill that in to get you much higher than 5% resource utilization and so because you'll be multi-tenant you can look at an app and/or I'm sorry at a node and understand what are all the different jobs and services that are being serviced by this particular node there's a lot more in the Nomad UI I only had time to show off a few of its features it's available now so please download it play with it and give us any feedback you have on how to improve it another thing we talk about all the time with Nomad is this separation between the concerns of developers and the concerns of operators so from an operator side we'd like to be able to stand up this cluster make sure has enough capacity run the developer jobs that it's healthy and available but then we want to be decoupled with from what the developers are trying to do whether they're trying to submit a new job whether updating a version of a job scaling up scaling down rolling back what is that they're doing we don't necessarily want to be tightly coupled in this capacity as being an operator and so while the UI certainly helps with this makes it easier for the developers to self-service and not necessarily have to understand the whole system it raises this other question which is how do we do this safely at scale right it works if we trust all of our operators trust all of our developers but as we open it to a much broader community inside of our organization how do we sort of do this without losing sleep so to make that a lot easier today we're also announcing nomads ACL system this is building on top of what we've learned with councils ACL system and vaults ACL system and so there's a few a sort of twist it's a little bit different from both of them but sort of conceptually very similar at the heart of it is a set of role based ACL policies so we can define what our roles for our developers what our roles for our operators wear roles for our DBAs and with each of these roles to give out sort of these very fine-grained capabilities so we can decide maybe our developers are able to submit a job and update a job but not able to access the logs that those systems are generating because they might contain PII data or sensitive data being logged out to disk so you have this very fine-grained capability system that lets you do lout exactly the kind of permission a developer operator needs to interface with the system those then get tied back into ACL tokens which can be used to make a request to the system and so because we're announcing the UI with it at the same time this is natively integrated so if you're using the ACL system with the UI developers can put in their credential and just interact with the system like normal and the UI will take care of threading through the credentials correctly so that's all we have for a nomad ACL system there's a lot more detail of it if you want the gory guts all of the documentation will be up today and all of this is available as part of the Nomad zero 7 beta this includes both the UI as well as the ACL system and a lot of other features that is available today something we talked about for the first time last year was what we call the Nomad billion container challenge and so when we were designing Nomad initially we really wanted to make sure that this is a system that would scale with our users that they don't have to architect around it so when folks like gedcom go from 10% traffic to 100% traffic they have to have that moment of oh shoot now we're reacting around the core element of our system and so what we wanted to do was design it with a branch mark a sort of stress test in mind that we thought was obscene enough that nobody would ever hit it and so we're like million containers no one's ever gonna need anything more than this let's said this is our design benchmark and so we partnered with our friends at Google they gave us access to 5,000 machines on GCE and said run wild with us and so we submitted a thousand jobs each with a thousand Redis containers and so I think you know at a million Redis instances we might comfortably have run the world's largest Redis cluster and what we found is that we could actually schedule all million of these containers in under five minutes and so this was you know great we were super excited that we hit our design threshold we're like we never have to worry about it again mission accomplished and then we got a call from Citadel group who's here they'll be speaking and they're like that's cute we need to go even bigger can it do it and we're like you guys are making our lives difficult here and so this was really exciting for us too because we on one hand were like well no one's ever gonna hit this and then on the other hand it was this really nice validation that it is useful there are people who are operating at the scale and need this and so what we're really excited about today is the availability of nomad enterprise so really looking at how do we support some of these enterprises that are truly operating at this gargantuan scale and looking at their unique problems so one of the things that became very clear is access control is needed by everyone at sort of any scale even if we're operating a very small cluster we need access control so we can safely move away from the assumption that being on our network means you're authorized to do things right you know it's part of our ethos around things like vault the network access should not mean authorization but going even further than that what we've heard from some of these organizations is they're not really one organization there are actually many different lines of business sort of housed under one roof and so they need to go even further than just having an access control system and have native ability to multi tenant a single shared cluster so as part of that nomad Enterprise as the support for native namespacing so now you can have separate namespaces for lines of business a B and C and each of these things looks like a virtual cluster in and of themselves they have their own sort of view of the cluster and so this lets them much more easily multi-tenant li share a single cluster now it's not enough just to split it into what looks like a virtual cluster because you still have to worry about quality of service you don't want one line of business to consume all the resources of this shared cluster and starve the other lines of business of resources so attached to those namespaces is the ability to quota each namespace to a specific amount of resource utilization so this way line of business a can consume their resources and do it safely without impacting line of business B's ability to do what they need to do and all of this is supported with with nomads native multi-region support so whether you're using nomads just to submit jobs across multiple regions if you enable ACLs we replicate policies and manage those globally if you define namespaces or quotas those are all managed globally and support the multi regions set up of nomads and so that's all we have for nomads at this point I'd like to welcome Mitchell Hashimoto back onto the stage thank you guys so much [Applause] [Music] [Applause] thank you very much Armand those were a lot of exciting updates a lot of great things consul going 1.0 a UI available for everyone and nomads really really exciting stuff and so I hope what you see is that every product is growing and maturing faster than we've ever been able to do it before and we're really excited to been able to share with you some of these exciting new features and so what we're seeing really is that the growth of infrastructure and applications has been really enabled in a big part by the ability to automate more effectively going on this you know undated timeline back to configuration and code and the rise of virtual machines those things enabled fast automatic machine set up that we didn't have before and then cloud providers infrastructure is code and tools such as terraform are enabling fast automatic infrastructure creation that wasn't really that was difficult to achieve before trends like containers and micro services allow us to much more easily package or application runtimes and get them out there and then schedulers have enabled automatic application placements deployment as well as operational tasks such as rolling deploys and so on that have allowed us to just deploy a much larger number of applications and so automation has brought all these really amazing capabilities that were simply not really possible before however it's also brought a number of challenges for example in the age of VMs and manual procurement if you place an order for 5,000 new machines a human would probably call you and say like are you sure this is the correct order or if you try to deploy an application that was requesting all the resources of your biggest machine again someone would probably stop you and say like is is this what you truly meant to do but today with automation these things could just happen and they happen really really fast and so today we we want a way to enforce problems such as forbidding changing configuration outside of business hours this is a pretty easy way to introduce instability page people have people work at times you don't really want them to be working it's something you want to check or ensure all services have associated health checks we want to make sure that every service we deploy can be monitored at least at some minimal level before it's actually running and serving user traffic or verifying that all the TLS certificates that we issue are encrypted with a key size of at least 2048 bits just to ensure some minimum expectation of security or perhaps to ensure that all instances we lodge in scenarios where we have multiple lines of business have a billing ID ID tagged with it so we know which organization is using this and these are all problems that previously could have been easily solved with human verification but today is very very difficult in the age of automation Institute so today what I'm happy to announce is the availability of something we call Sentinel it's a system and framework for policy as code Sentinel is an enterprise only feature that is being integrated into all our products and the CL Sentinel solve some of these problems let's take a look at those examples one more time so each of these examples actually corresponds with a Hacha Corp tool that's capable of automatically solving that problem in combination with Sentinel and going through each one if we take a look at the first one of console enterprise we'd like to forbid changing configuration outside of working hours and this is actually the Associated Sentinel policy that would enforce that we're gonna cover language syntax and other things a little bit later in detail but I hope what you could see here is that it's pretty easy to understand and see what it's doing which is verifying that all keys that start with the service namespace can only be modified within the standard business hours of 8 a.m. and 5 p.m. and if we take a look next and nomad we want to ensure all services have associated health checks equally easily we're just looking at the job structure and verifying that all tasks that are within the job structure have at least one health check associated with it moving on next to vaults where we'd like to maintain some base security by ensuring all TLS certificates are encrypted with at least a 2048-bit key we do this by making sure that when a PKI roll is created which is how you define who could create TLS certificates within vault that their minimum required bit size is always at least 2048 and then finally with terraform ensuring all instances have a certain tag that we expect this one we could go over a plan that's been made and verify than any AWS instances in the plan contain a billing ID tag these are for common problems that real customers have brought up to us over the years and they all now share the same easy solution and that solution is what we call Sentinel and so those are the problems The Sentinel can solve but what is it how does it work let's take a look at that next so Sentinel embraces this concept that we call policy as code and what policy is code does is bring all the same benefits you know in love with the infrastructures code and bring them down to policy and by representing policy as code we're able to use proven software development practices with policies and this is in contrast to other policy systems that use graphical user interfaces for updating or application specific endpoints because the problem with these is that they're very difficult and not easily repeatable version Abul or most importantly testable and Sentinel solves all of these cases and to do that Sentinel is really four separate things it's a language it's an embedded framework it's a workflow and it's an ecosystem of imports and so starting with the language Sentinel does views its own policy language that was designed specifically to make writing policies easy we designed it with a with hundreds of customer examples and translating those from English into the policy and trying to make that as easy and straightforward as possible one of the things we found was that a lot of the people that are responsible for enforcing this policy within businesses do not have a strong programming background you cannot throw at them a programming language because they're not comfortable doing that but with something like Sentinel where it almost becomes English and it's almost 1 for 1 they were able to effectively write and enforce policy across multiple systems it was also built to be safe in a highly security sensitive environment such as vaults Sentinel is integrated with vault and we can't compromise the security of vault because of it and so that was designed with that in mind and then to run this language sentinel is an embedded framework and embedded is a really important word because what that means is that sentinel itself is built into the applications you don't run another application next to it you don't have to deploy something new and sentinel is actively right in the data path of what's happening and so what this lets you do is what we call active enforcement sentinel actually actively sees a behavior coming in before it takes effect and could perform policy checks on that behavior at that time and reject them if they fail so you don't detect after a keys been written to console you don't even allow that key to be written to console in the first place the policies run right there but while we do active enforcement sentinel is also capable of passive enforcement so sentinel can continuously run in the background and continuously check the sate of your of your systems to ensure that they haven't been pushed into a policy violating behavior by some external force so for example with terraform we could continuously run sentinel against the terraform state and if someone goes into something like the AWS console and deletes a tag we will detect that your system was pushed into a policy violating behavior outside of the outside of the terraform workflow sentinel also supports natively supports the concept of what we call enforcement levels and in and enforcement levels of supports we call advisory soft mandatory and hard mandatory enforcement levels are broad types of policies that we found existed across all of our customer advisors while building Sentinel advisory policies allow a policy to fail but actively show a user a warning of what failed and why these are useful for scenarios where you'd like to educate a user that they're perhaps doing something not quite right or if you're deprecating something and you want to warn them that they shouldn't be using this version anymore soft mandatory policies require that the policy pass but it could be overridden with the proper privileges these privileges may be the same person they may require another person and these these policies are super useful in cases such as don't allow deploys on Fridays deploys on Fridays easy way to get people working late or over the weekend maybe that's not a good idea but of course you have to deploy sometimes on a Friday things happen so soft mandatory policies are way to require an explicit override and explicit acknowledgment that you're doing something that could be dangerous and overrides always appear in logs so that you know when overrides happen who initiated the override and who initiated the original requests as well and then hard mandatory levels cannot be overridden and must pass under any circumstance short of actually removing that policy itself which itself is a fairly privileged operation and hard mandatory policies are really great for business requirements as well as legal compliance and assisting and making those a little bit more robust and so we have the language and we have the embedded framework to actually run that language and enforce policy and we think the sentinel is a really great technology but having a really great technology just isn't enough and doesn't solve the actual core problem that we're trying to solve you also need to think about the development test and deployment workflows of these policies we think those are just as important as actually building a powerful policy system and so what we've done is build into Sentinel a powerful workflow so Sentinel ships with a local CLI that we call the Sentinel simulator The Sentinel simulator has is a single binary with zero dependencies and it lets you write and test policies against any Sentinel enabled system without access to that system all locally on your own machine the policies are developed in text files we are pushing policy as code so you'd code your policies and text files and they could be run using a command called Sentinel apply this command can mock any data that a system exposes to your policy so that you could develop that policy without that system and when policies fail you could also see the failure but also the logic that led to that failure and the various rules that didn't that passed or failed the led to that result in addition to simply developing policies evaluating them locally seeing if they pass fail we expect all our customers and users of centinall to eventually grow to hundreds thousands of policies and reasoning about those insuring they behave correctly is extremely important and is a challenge today so Sentinel also includes a full test runner and test framework within the Sentinel within the Sentinel simulator that you can use to test your policies and what this looks like here is you define an environment in which the policy is running again you mock all the information that the system would give you you don't have to run console you don't have to run vault it all just runs in memory and what you do is you assert that the policy behaves as you expect but you don't just assert that the policy passes or fails because perhaps that policy is passing or failing for some completely unrelated reason from what you're testing so you could also assert the various logical rules that led to that pass or fail so that you know it's passing or failing for the reason you expect and this CLI was designed to run in continuous integration environments so you could see no parameters as long as you follow the right folder structure will run thousands of policy tests automatically all in memory and we're encouraging everybody to run this in there CI environments to constantly have continuous testing of their policies this also lets it work extremely well with version control pull requests for policies etc you get all those benefits directly with Sentinel that you can't get when you're manually editing policies directly through a system and so that is the powerful workflow Sentinel provides and the last thing I want to talk about is the ecosystem of imports Sentinel can use input plugins that we call imports the source external information for use in policy decisions and this is a really extremely powerful concept especially once you think through the ramifications of this your policies aren't limited to only the information that the system is giving you to make that decision you could query and get information from any external system you want in order to make your policy decisions so as an example we have a customer that uses terraform they also use a service now that wants to ensure that their terraform changes can't go into infrastructure until there's a change management request in ServiceNow that matches that change and they could do this with sentinel by writing a plugin these plugins are exposed to sentinel as imports and imports expose functions and data you could use within sentinel to make these decisions and the sentinel simulator can also mock all of this data so you don't even need access to the plugins you don't need to have access to a ServiceNow cluster if you're writing that plug-in you can mock all of it locally to still develop policies against this right on your own machine offline and anybody could build a test Sentinel plug-in using the SDK which is publicly available the SDK provides a test framework to verify your plugins and gives you examples in order to build these things so that you could work with any external information source and these plugins then work with any sentinel enabled application so for example if I wrote a plug-in that source data from s3 the moment I build that plug-in I can now source data from s3 in console policies Nomad vault terraform all of them without modifying that binary the plugins use our production ready plug-in system we've developed for the past five years and build on top of it and additionally the plugins use the new G RPC mechanism and our plugin and our plug-in framework so you could write plugins and any language and those are the four aspects of sentinel that make it a really incredible system for policy as code and as I said earlier sentinel is integrated natively into all our enterprise tools it's available in the next version of each and it's ready to go I'd like to show you what it looks like in each of those so with terraform enterprise you could define policies that run between a terraform plan and a terraform apply and these policies have policies have access to the entire terraform configuration the terraform plan the tariff forms state and a simulated version of the state after they apply so you could use all of this information to determine if an apply is allowed to happen and so this is the same example as previously but you could see we're going over the plan to verify it has a tag equally you could verify over the plan to verify for example that no tags are being removed that only tag to be added you can't remove things you could do just about anything with this with console and console enterprise you're able to define policies that run during kV modify events as well as service registration and updates and so in these scenarios you could enforce whether a kV modify is allowed to happen or service registrations allowed to happen and so in the example I have here what we're doing is if the service name is prefixed with a user and a hyphen we're enforcing that that must be deployed within a cider block that perhaps has higher levels of security or monitoring or something we just required those services to live in that cider block and we can enforce that right here with vault enterprise policies are split into two types of policies we have what we call role governing policies and we have what are called endpoint governing policies role governing policies are attached as you might expect two roles so whenever you make an API request using a token that has that role we execute that policy endpoint governing policies are attached to specific API endpoints whether you're authenticated or unauthenticated and whenever you hit that endpoint we execute that policy both of these policies have access to the same amount of data which is a lot you have access to the full identity of the person requesting you have the full request package and a lot more and so in this example what we're actually doing is ensuring that people who log in with LDAP are only allowed to request read only database credentials you must log in with another mechanism in order to get right database credentials and then with Nomad enterprise you're able to define policies that run before a new job is created or a job as updated and these have access to the full job definition that's being sent so what you could do with this in this example for example is ensure that all tasks all jobs that are deployed into the nomadic cluster are sourcing their artifacts from an internal artifactory instance they cannot source artifacts from an external source and you can see sort of where you could go with this and the amount of power you actually able to have and one of the customers that we worked with in the design from the very beginning with Sentinel was Barclays one of the world's largest financial firms Barclays is planning on using Sentinel to provide safety guardrails for provisioning cloud resources this will allow their expert operators to define reusable tariffs or modules within the registry best practices with terraform and give them out to over 15,000 users of operators within terraform and know that they're safely using terraform across their entire organization it's really great to see how Barclays is gonna put sensible to use in their cloud and we have to thank them for being really great customer advisors during disease during the design of Sentinel and so we're really excited about Sentinel and you can learn about it today at Hasek or comms slash sentinel and with that i'd like to invite to the stage our CEO Dave McKean [Music] Thank You Mitchell and and welcome from me it's as Mitchell mentioned it's actually our third hacci comp but what he didn't mention is it's almost exactly five years since the founding of Hasek over itself so the velocity of adoption of our products is actually been kind of amazing for you know a relatively short period of time to see the number of people to put up their hands when we asked how you know who was running this in production it's a big number but what we really saw over the last 24 months is almost sort of a step function shift in adoption not just by end users but also by partner ecosystem around us and you saw that from some of the announcements that came out today where our products are being used not just by some of the largest entities in the globe to run their own infrastructure but also increasingly underpinning sort of foundational elements of the of the cloud ecosystem which is really really fun for us but it's also something we take super seriously and so as a company we've been for the last 12 months investing super aggressively and we'd be really investing across basically three vectors one of them is around the product and engineering organization itself where our engineering organization as you can see from 150 releases that we put out over the last 12 months of our core product portfolio has more than doubled in the last 12 months and it'll continue to expand if not double again over the next 12 to 18 months secondarily we recognize that a lot of our users are served navigating this transition from a relatively traditional way of running infrastructure to a more you know cloud oriented approach for running infrastructure it may not always be on cloud infrastructure but philosophically it's a slightly different way of running infrastructure and so we've invested really really heavily in our support and successor organization to help customers navigate that transition and the third and this has been a really big push for us is we recognize that we need to be with technical expertise where our users are so we we over the last 12 months have established presence really all over the globe we now have technical resources on the ground on the East Coast on the West Coast and the central in the federal area I was in London last week we have I believe eight people in in the UK today we have in Japan and Australia we also have customers in Korea and that is something you'll see us continue to do because this in our view is sort of a sort of a generational shift of how infrastructure is deployed and run and so we need to be where our customers are not just not just sitting remotely from our offices here I think you know actually a good example of this we caught my attention that the London meetup group alone for Hajj Corp is over 1,500 people right so it's not even affiliated with us necessarily but it gives you a sense for the stuff is used everywhere it's really the engineering organization and investments that have allowed us to to make the announcements that we made today and I just wanted to play them back because as you see we believe strongly that there are four essential elements of infrastructure right there's a provisioning layer there's a security layer there's a runtime layer and then there's a need to connect everything together so what we announced today was actually updates across the entire portfolio on the tariff front side we announced the terraform module registry which as I can tell on Twitter and on hacker news is already super popular I think it's going to put some pressure on our operational teams to match it also the newer version of terraform Enterprise which adds collaboration and governance capabilities to the to the running of terraform in a large organization as Mitchell identified secondarily we announced vault kubernetes support which has been a real request we realize there's a lot of heterogeneity across infrastructure stacks and it's important to us that we enable people to run the technologies of their choice urbanised has been an important one for us constant 1-0 and no no bad 0.7 or huge and then we introduced sentinel as a really sort of a governance model to apply across all four layers of our stack and while the company's changed a lot over the last it's called 24 months the philosophies that underpin how we think about the world really haven't right Mitchell and Arman started with has started this journey with the view that enterprises are heterogeneous things yesterday it was virtual machines tomorrow tomorrow it'll be you know of today it's containers tomorrow it'll be something else and so what we do instead is we focus on the workflows not the technologies you see that in our in our approach and philosophy with respect to terraform where not only has the provider ecosystem expanded as it relates to support for more and more cloud infrastructures for example you know you saw some of the ones today but basically across any infrastructure type that you might choose to provision there's a terraform provider support but more interesting to me actually is there are now more than 70 terraform providers and we have a queue that we're certifying another ten behind that because as you all in the room have to use tariffs or and provision infrastructure to your cloud partners you know the IFV vendors that that you work with and must be included in that provisioning experience had stepped up and created terraform providers to allow their their products to be provisioned as part of that single experience it is that focus on on the workflows not the technologies which is just underpins our whole philosophical approach the second core principle is around automation through codification that's a belief in immutability and the need to codify given the scale of the of the environments we all operate Sentinel is a good example of how we are now applying the governance model for distributed infrastructure through codification Sentinel also includes I should add the ability to extend through the to the plugin mechanism to whatever back-end system you may want to connect to that experience so the philosophy around openness and extensibility recognizing that you need to be able to extend our products to your specific environments which are highly differentiated as any large company is those remain the underpinnings for how we think about philosophically our approach so I know it's been a long day and I we certainly honored that you chose to spend a couple days with us we know it's a big deal for all of you to take your time away I wanted to call out despite the fact that they were really really rich agenda I just wanted to call it a couple of things there are deep dive sessions on Sentinel the module registry and also on all the new product announcements that we made I also make would make sure you you you attend our evening social tomorrow as well as the the hub next door I know a lot of people were there last night so with that thanks for coming today we hope you have a fantastic couple days thank you very much [Applause] [Music] hello hello again look at your people trying to leave sit down thank you very much hopefully you've enjoyed all of the announcements in this morning's keynote but I do need to clarify a few things you may have noticed that the first two talks are not across all three tracks they're only in track B and track C and they were a little bit obscure on the schedule and the reasons are because they correspond to two of our major announcements this morning so if you're interested in Sentinel Nick Jackson from Hacha Corp will be talking and doing a deep dive with demos on Sentinel this morning in track B in track C we have folks at grunt works who are talking about the terraform model registry and terraform modules in general how to build reusable terraform modules there will not be a talk in this room for the first session so tracks B and C are the only tracks that are running we do ask that everyone leaves this room because we have to prepare it for the talks of the remainder of the day so after we're done please do exit we have a short break so we have about 20 minutes the next talks will start at 11:45 please make your way to the hub or other seating one last thing as you may have noticed this room is a little full we had to bring extra chairs in this morning there will likely be standing room only for tracks B and C so if you're interested in attending those please get there early and we apologize in advance if we are not able to get you a seat but we're working on adding more seating to those rooms thank you very much we look forward to seeing you through the rest of the day thank you [Applause] [Music]

Channel: HashiCorp

Views: 6,203

Rating: 4.878788 out of 5

Keywords:

Id: b6nn7vLdjo8

Length: 94min 11sec (5651 seconds)

Published: Fri Oct 13 2017