Keynote: The Tao of HashiCorp - Mitchell Hashimoto, Founder, HashiCorp

all right thank you very much it's a huge honor to be here it's really cool to see talks like those given by the banks because as many of you here know my life currently is heavily involved with open source but maybe sort of lesser-known is that I was really only able to start getting involved and learning to code because of open source so as a quick sort of story leading into this talk I started to learn to code around 11 or 12 years old and we had one dial-up line and I was not allowed to connect to the internet during the day because my dad needed that line so I really only had two ways to learn to code and one way is super not open-source at all my dad would drop me off at Borders bookstore which I think is now out of business but what dropped me off at the bookstore I would run to the corner where the computer books were I would like crack open the book to not crease it because I couldn't afford to buy the book so I would crack it open read and read and read and I would read the same like five pages over and over because I had to memorize them because I couldn't take this book home and then I would run home and then try to sort of apply what I learned and I would repeat that sort of weekly and then the other thing was when I discovered that open source existed and you know github doesn't even exist Google was relatively new so I tried all these different search engines but he eventually found out you know people uploaded tar balls and stuff of code and so I would download a bunch of tar balls at night and then during the day when I couldn't dial up I would just read the source code and learn sort of how to apply these abstract concepts books are really great at abstract concepts open source is really great at applying those concepts and showing me sort of how it could work so that's how I really got started and and so it's very fitting and I'm very happy that I'm able to sort of build my career around open source software and around eight years ago I started a project called vagrant and as the bank said I'm talk it wasn't my first open-source project there was many before that that no one knows or cares about but vagrant ended up being something a little bit more special becoming a lot more pop than I could have imagined but also taught me a lot about you know community the non-coding side of open-source I had to learn about the challenges of building a community the challenges of bringing on contributors governance those sorts of things and the success of vagrant and and getting into DevOps is also what led me to found Hoshi Corp which is a company aimed at building striving to build you know the best infrastructure automation tools and I started this company with my best friend Arman and together we've built seven free and open-source license products that we've built our company on which do development environments infrastructure automation cluster scheduling sea security software there's a pretty wide array of things and when we started Hoshi Corp we knew for sure we were building around four or five of these things and we started that process and when we started it was just the two of us and when there's two of you and especially when you're as as close as we were it's it's easy like you have a mutual vision and a mutual understanding you know how you want these things to work how you want them to feel and you could be really really productive but we knew that to really you know reach the goals and ambitions that we had for a company like Hofstra Corp which is to make a much broader industry impact that it was gonna be more than the two of us and very likely much much more than the two of us so our biggest fear was how are we going to scale or not not company vision but product development and design vision like how we're gonna build an engineering organization that really understands why we do some of the things the way we do some of the things we do not necessarily just what the problems are solving but how we solve those problems and so very early on in the company and two most very surprisingly early we wrote a document called the Tao of Hoshi Corp I think we wrote it when we were still around five employees so it was really early we published it about a year later I think we published it in 2014 but I'm not 100% sure if that's right but what the Tao is is really sort of the principles that guide sort of our vision or roadmap and product design of our products and when you know I've worked in companies where when I hear the word principles I think it's some HR checklist that the company just has and with something like the Tao I think it's it's important to know that we wrote it when we're five people it's important to know that we've applied it really rigorously sort of to our tools and so what I want to do in this talk is go over the principles of the Tao explain what they are why they matter to us and how we've applied them to our tools and my hoping with my hope with that is that you could take this and more broadly applicable to not just building tools but also a design decision and and when you're adopting tools like is adopting this tool is our certain element so that's how important to you and so an architecture you know issue with this tool is not going to work for you and yes so let's get started going through the town there's eight of them a few of them are really similar so they'll be very fast and some I'll spend more time on so the first thing in the tao is workflows not technologies and it's not a mistake that we put this one first it's really really important to us and so the idea behind this is that technology is both software and hardware continue to shift like continuously will shift if you expect that anything is going to remain constant you're in for a rude surprise and so with this idea the the core thing is that product design should start with an envisioned workflow to solve a problem and so I use a few words here I say product design and I'm probably gonna say that a lot when we start a new tool whether it's a new open-source tool or a new internal sort of back-end service we don't start with directly with algorithms or architecture or things like that we ask ourselves what's the problem we're trying to solve and then we start designing and and I use the word design not in the word in the meaning of visual design but actually holistic design of how is it gonna feel but also how is it gonna work and so the examples I have here from our industry are sort of looking at you know mainframe to server server to VM VM to container or if you look at physical data centers to cloud all these are huge paradigm shifts that have happened sort of over 10 to 15-year periods and in all of them lots and lots of software was replaced and if you look though the core problems that people are trying to solve just didn't really change and so the idea behind this is that most problems don't disappear some do but most problems don't disappear some problems are created of course but there's a big group of problems which stay the same and so if you work if you aim at work clothes first you could often create software that could span multiple paradigm shifts and that's really important to ease the adoption and transition into these different different stages and so the best examples for our software here are terraform and vagrant terraform is a tool for creating infrastructure with terraform you could create VM so you could create physical servers you could create containers you can manage SAS applications and we have people from hobbyists all the way up to you know the world's largest on any given day company I'm using terraform to manage vastly different things but using the exact same workflow and so the last thing I'll say about this against really importance of spending a lot of time on this slide but is is this is how we also start our software terraform and vagrant actually both started as bash scripts and when I say that people usually think I built like a Minimum Viable Product and bash I didn't I just put a bunch of echo statements and pretended something was happening when nothing it did absolutely nothing but the reason I do that is I is I write it in bash and then I use it for a day I use it given it does nothing but I use it and try to see if it feels right like is this how I actually want to work on a day-to-day basis because I'm gonna commit this sort of like building some that I'm gonna you know work on and use for years is this how I want it to work and that's really the focus on workflows the next thing much simpler much more obvious I think is is the concept of simple modular composable this could kind of be described as the UNIX philosophy in a way but I think that's that the UNIX philosophy is always interpreted to the speaker's like you know the most beneficial definition so instead of saying that the goal of this is really just to prefer smaller components with well-defined scope that are functional on their own but can be integrated and composed with other tools to do new and interesting things and I think this is really obvious in our and in our in the ethos of our company we are sort of one of the few startup companies that built so many pieces of software that dude vastly different things instead of bolting features on two different products vault is a good example we built vault because people were putting secret information in console which is a key value store and we didn't feel console console security story it's it's just not built for that and so we felt sort of guilty in a way that people were putting secret information in there and we didn't have a better option to give them we could have bolted on encryption to console and certainly from a pragmatic standpoint that is the way we started the design but as we started thinking through the the Turtle tower of security problems you know beginning with where does the key stored how do you rotate the key who has access to the key how has security bootstrapped you know as we went through these turtle problems we realized that it would be a humongous problem a solution to just bolt on to console and would be way out of scope where that project was trying to do and so that's an example a building vault the next thing is communicating sequential processes so if you're building these modular components just built on modular they have to be able to communicate with each other and for that we just follow this model where all our software should be a standalone process that runs well on its own with a well-defined sort of communication interface an API that they use that they used to compose each other I think that's fairly straightforward the next thing is immutability and so immutability is of course the idea that things should not be able to be changed and the most popular and and very appropriate for this conference example is version control right with things like get we we get the benefits of immutability which is sort of the idea of atomic you know it's changeable with some surgery but these atomic units that represent a version and in a history behind that and who did you know a sort of blame trail and these are only possible if you can't just change things whenever you want in the middle and so immutability is really important to us because these concepts are really beneficial to other things and a good example is infrastructure if you consider sort of each stage of your infrastructure as an atomic unit that's driven by a specific change then you could see the history of your infrastructure you could see who impacted that history when the change was created but you could also reason about desired states current state and sort of how to get between them if you know that what you should look like is exactly this and not this with these little changes it's exactly this and you're in this other world view you could transition to that and that's really important for infrastructure next thing the next two are really the idea of codification so codification is the idea of encoding knowledge as code and so with infrastructure historically knowledge is passed on via oral tradition you join a company in there you know assisted min or IT or office or whatever group you look at the infrastructure and you ask you know why are things network this way how does you know from from a web user to this service how does how do what what applications that go through how do things work and the most common response has historically been oh for that you should talk to this person and for this you should talk to that person because they're the subject matter experts on that and the problem with oral tradition is is it's slow it's also just thick because human memory is fickle and so you don't quite get you know the truth usually and so we ascribe very heavily to the idea that knowledge should be represented in code and use that as a source of truth so with tools such as favorite for example how do you create your development environment the answer before used to be read this humongously large read me and follow this series of steps and the answer today can be we have this code that is also executable like the document and description of what does your dev environment is also the configuration for the creation of that environment and so that is the second thing which is automation through codification the idea that manual processes should be encoded as code and automated and this in our industry is just necessary and I heard talk yesterday talking about just growing scale and when you're this this sort of scale of infrastructure growth and the scale of application change in the data center has already exceeded the point of human capability and so we need atom a Shahnaz a tool to leverage ourselves to be able to manage that and if it from the talks I saw yesterday you know if you imagine a world if you believe in a world where there's only gonna be you know an order or more orders of magnitude more devices whether they're whether they're my refrigerator or I really don't want my refrigerator to ever be connected to the internet but whether it's my refrigerator or you know a car or a watch or whatever it is this is just going to put more and more burden and so we're gonna need automation to be able to scale these processes properly so automation is absolutely critical and the vehicle for automation that we choose is codification which is different than simple scripting which is the type of code but it's this idea that sort of the knowledge and processes is the same document that is automated and the next thing is the idea of a resilient system so we're deploying more and more things we're moving more and more to cheaper less reliable whether it's hardware or heart uh you know infrastructure providers and distributed systems smaller devices things like that and so in building this systems have to be resilient we're it's not a matter of you know if there's going to be a failure it's a matter of when that's always been the case but the when happens to just happen all the time nowadays and large enough infrastructures and so you have to consider resiliency and there's various approaches to resiliency but the approach that we like to take is this desired state model it's the idea that to build a resilient system the system needs to be capable of understanding a desired state it needs to be capable of determining its current state and it needs to be capable of creating a plan or a path to get from its current state to the desired state and if you consider that abstractly that describes leader election systems that describes infrastructure configuration and desired state that describes configuration management this is sort of how we build resilient systems and if you could build this property into your software the idea is that they all should strive and automate to being in the correct state all the time and so you can see examples of this with Tara form which is Tara when you run terraform there's a plan sub command and when you run plan what it actually does is inspects the current state inspects your desired state and shows you what it would do to reach that because changing infrastructure is scary so it shows you that before you could apply it with things like consul leader election like I said could move to is the desired state of having a server a leader and then the last element of the Tao which is really my favorite but also the one that has actually been the most challenging for for some people that have worked at our company we've actually had some people leave the company which which have left because we're just we're just too pragmatic but it is pragmatism and pragmatism is the understanding that everything I said before is a set of ideals and their ideals that we should aspire to and the deals that we should fight for and build into our tools but if it's just simply not the right pragmatics then we need to be able to be understanding enough to reevaluate those ideals for the given problem set at hand and so a good example of things like this is that not everything we do is immutable immutability introduces a number of challenges and immutability is not always practical and so the biggest offender of this is consul is our kv store consul doesn't actually have versioning sort of built in doesn't have this concept of immutable change sets it has the idea of transactions and things like that but but to some people that's not enough and so they'll reference our Tao and say you know you're you're violating this principle that you had which is fair but at the same time it's it's a practical it was a pragmatic decision at the time and we certainly you know got built features around that more and more as time has gone on now so pragmatism is absolutely important when building these building against ideals and so that is really the Tao of pascha corp and and I hope that if you've used our software you could see it every day and where you're using all the tech leads or projects all the engineers of our projects continually sort of apply this on ayane RFC by RFC basis when new features are being developed but this is also the principles that we use when evaluating software that we don't write when there's five different choices out there we tend to choose the choices that align closest to our ideals and by having them written down and having them clear you know our opinion isn't isn't changing and isn't isn't just what's best for the moment it's actually really clear of what we care about and what we're looking for so I hope this helps thank you [Applause]

Channel: The Linux Foundation

Views: 4,845

Rating: 4.9555554 out of 5

Keywords: technology, quantum computing, cloud computing, linux, nfv, open community conference, IBM, open daylight, containercon, Open API initiative, unikernals, API, decoding, Cisco DevNet, CPU performance scaling, technologists, linuxcon, sdn, system containers, containers, containerization, Intel, open source, openstack, kubernetes, red hat, open source summit, cloudopen, apache spark, open source community, Google, openvswtich, devops, docker, embedded systems, cloud

Id: YZXngLn5UJk

Length: 19min 41sec (1181 seconds)

Published: Fri Oct 27 2017