[MUSIC PLAYING] SETH VARGO: Hello, and everyone. [CHEERING] Welcome to the final session
of the day in this room, Using HashiCorp Vault
with Kubernetes. My name is Seth Vargo. I'm a developer advocate
for Google Cloud. You might recognize me from
the "DevOps versus SRE" series, where we say, "class
SRE implements DevOps." I'm excited to be here today. Prior to Google Cloud, I
actually worked at HashiCorp. I worked for our other
speaker, Armon Dadgar, who is the CTO and
cofounder of HashiCorp. And I will bring him onstage in
a minute to talk about Vault. Just a quick overview
of today's agenda-- we have four distinct sections. You'll recognize
them by the colors that they're
associated with here. We're going to start
with a quick overview that Armon is going to give us. I know some of you might
be familiar with Vault, but we want to make sure that
we have a common baseline. Next, I'll tell you how
to run Vault in Kubernetes as a service. We're going to get
really technical, so I hope everyone's ready. There will be a live demo. Then, I'll show you how to
connect to that same Vault cluster using
Kubernetes applications from different pods and
different services, again, accompanied by a live demo. And then finally, Armon
will talk about some of the new and exciting things
that are coming on the Vault roadmap in the future. So with that, I'd like
to welcome Armon onstage. [APPLAUSE] ARMON DADGAR: Thanks
so much, Seth. I'm hopeful we will get
through this session without running out of disk
space, but fingers crossed. So like Seth said,
what I was hoping to do was, for folks who are
less familiar with Vault, start off with a bit of a
high-level introduction. What is the problem motivation? I think we're probably all
a little tool-overwhelmed. So hopefully, there's a
good enough reason for us to consider adding yet another. So before we really start
talking about Vault, I think it's useful to
talk about the larger problem, right-- secret
management as a whole, which is really what Vault
attempts to solve. And so when we talk
about secret management, I think the first question, kind
of the most natural question is, what exactly
are we managing? What is a secret? And so one of the things we
talk about is, what is a secret, and how is it different
from sensitive information? So when we talked about a
piece of secret information, it's something that allows
us to either authenticate or authorize against a service. So an example might be
a username or password. We can provide that to, let's
say, a database to authenticate a session against that
database, prove our identity, and then the
database is allowing us to request data from it. Similarly, something
like an API token, we can present that when
we make a call up to, let's say, Google Cloud to read
and write from Blobstore, it's authenticating the client. It's authorizing us to
actually do that request. And finally, something
like a TLS certificate-- and you might not
log into a system exactly with a TLS certificate. But you're presenting
it to a client, and that's authorizing you
as, you know, let's say, www.hashicorp.com,
and so it's allowing us to sort of elevate ourselves
into this trusted position. So any of these types
of materials-- username and password, certificates--
are sort of secrets, right? And the reason we want
to keep them secret is, if they fell into
the wrong party's hands, they could use it as a
mechanism to authenticate themselves or authorize
themselves to perform actions they shouldn't, right? They are a mechanism of
access, and so those mechanisms need to be tightly kept. On the other hand, we have
sensitive information, right? This is a little different
because it's things that you'd like to
keep confidential, but can't directly be
used to authenticate. I can't use my customer's
social security number to log into the
database, but it doesn't mean I'd like that
piece of information on the public internet. So it's important to make this
distinction between sensitive things, which Vault
can help us manage-- and we'll talk a
little bit about how-- versus secret
things, which Vault intends to put inside of it. You use it as a database to
hold onto secret information. So when we go from saying, this
is what a secret is to, then, talking about secret
management, the way I like to talk about it is,
what are the set of questions we want to be able to answer? So when we talk about
secret management, these are the key ones, right? How do our applications
get these secrets? My web application boots, it
needs to talk to a database, how does it get a
database password? On the same hand, how do our
human operators get access? If I'm a DBA, and I need
to connect to the database to do some maintenance on it
or fix some issue with it, how do I get that same
database credential? Then, how do we think about
updating these things? We don't want them to be, sort
of, static credentials that live forever. We want it to be
periodically rotating them, whether for compliance
or just good hygiene. So how do we think about, if
I change my database password, how does that impact
my applications? How does that
impact my operators? How do I actually
automate that flow? Then there are the bad
things that could happen. What if this thing
is compromised? How do I think about
revocation of access? How do I do forensics
on who's done what when if I'm trying
to figure out, are we in a
compromised situation? Where has this key leaked? And so what do we do in
the event of a compromise, everything from that
forensic understanding, to reacting to it,
to rotating it? So these are the hard questions
that we'd like answers to. So the state of the
world, when we generally talk about secret
management as a problem is not necessarily great. What we often describe
it as is secret sprawl. This is a scenario in
which secret material is sort of sprawled all
over our infrastructure. It's hard coded in source code. It's in plain text
in our configuration. It's checked into
GitHub in plain text. It just litters everywhere. And the challenge with this
is it's very decentralized, and we have very limited
access controls around it. Like, who can access
GitHub in our organization? Probably anyone with
single sign-on, right? Who can see this plain
text configuration? Who knows? It's not something we're
tightly access-controlling. And so we have limited
access control around it, but also even worse visibility. So then, if we
actually wanted to do an audit of who all has looked
at this particular source file in GitHub, do we
actually have an ability to do fine-grain auditing
and forensics on that? The state of the world--
not necessarily great when it comes to answering
some of those secret management questions. And so this is really
where Vault enters. Vault's original purpose
was to try and look at secret management and say,
can we provide a good answer to these set of questions? And so our goals at the
top line are starting with a single source of truth. How do we move away from
this decentralized sprawl to a world that is centralized
and managed in a single place? And so part of this is really
understanding those two different kind of consumers. There is our
application consumers that want programmatic access. The web server is going to boot,
it wants to fetch a credential and do it in an automated way. We don't want a point
and click interface for that-- so API-driven
automated access for applications
doing automation. On the flip side, we still have
our human operators, right? They want something
nicer than an API. They want command lines. They want user interfaces. And so we want these
two access patterns to both be really
optimized so that we have the single source of
truth, and we want a solution that's cloud-friendly. So what that really means is
it should be easy to operate. It should be not dependent
on specific hardware that we need to deploy
in a private data center. It should be [? pure ?]
software that can easily operate in a cloud environment. So then, when we
translate that into, OK, these are Vault's
top-line goals-- how do we actually achieve
that, the sort of table stakes where things start, is just
a secure place to put bits. We want to put bits in,
have it securely stored, get those same bits out,
whether that's, let's say, database username and password. And so in-transit, when
clients are talking to Vault, Vault forces end-to-end
TLS unless you change it, by default, it asks
for the highest level of 1.2. And everything at rest
is encrypted and managed with a AES. So we have AES 256-bit, both
encrypted and authenticated so we can detect
tampering of data at rest. And this is exposed
to the end user as a hierarchical
key value store. So you can apply
some organization around folder structure
and grouping things together in a way that
makes sense for you. So to just make this a little
bit less theory, a little more concrete, here's an example
from the command line interface of both writing
and reading a secret. So what we see here, the
first command, is we're doing a Vault write
against a path, secret/foo. It's like I said, it's a
hierarchical file system. So in this case, secret is a
directory, and foo is the file, the leaf node we're writing to. And then we can
write opaque values. So we're writing, bar
is equal to bacon. But you could imagine that
would be username is Armon. Password is 1234, as an example. And then we can come
back around, use the CLI, and do a read against
that same path. So we're doing a read
again secret/foo, and what we see is we're getting
back bar is equal to bacon. So what this is really
masking for us is really thinking about, as a client, is
this encrypted over the wire? The CLI is talking over an
encrypted channel to Vault, hitting its API. And then Vault is
transparently doing the encrypt as it writes it out to disk,
doing the decrypt before it provides it back
to us as a client-- so doing the right
things, making sure we're doing this with
hygiene, but kind of making this easy as an end user. Just to illustrate the
equivalence of that, here's the same example as
what we just did on the CLI, but hitting Vault's API instead. So like I said, one
primary consumer Vault is our applications
or automation tools. And they want to consume it
programmatically with an API. So here's just a simple example
of doing cURL, where we're basically doing a write against
the path same path of foo and encoding the data
that we want stored, and then doing a readback
against the same thing. So it's meant to be a
simple RESTful API that's easy to integrate with. And so that's one level. That's table stakes is
putting our secrets somewhere that we know it's encrypted. We have audit logging and
access control around it-- all of that kind of good stuff. But there's still a bunch
of hard questions, right? There are questions like,
what if an operator connects to Vault, and they just
find it annoying to have to go to a Vault.
So they're going to copy everything out and put
it in passwords.txt so they don't have to bother to log in? Or an application starts-- and I'm sure we've never seen
this-- the first thing it does is write all of
its configuration out to standard out
or its log file. So it just reads all of the
database username and password, dumps that to disk, and ships
that off to our central logging system in plain text. And the third sort
of a classic case is we had a diagnostic
error, right? It's a 500 error on rendering,
or we're in a debug mode. And it says, here's my
database username and password, by the way, if this
helps you with debugging. So the thing you
find is the entities we're giving these
credentials to, whether it's applications or operators, do
a terrible job keeping secrets. So this kind of raises an
interesting challenge of, how do we still
protect these secrets, even though all of these
scenarios are very likely? So Vault's approach
to this, the way Vault thinks about
this problem is the challenge is we have these
long-lived static credentials. And the challenge with a
long-lived static credentials is if they leak anywhere, ever,
either through passwords.txt, or through a log file, or
through a diagnostic page, it's game over. We've arranged it
as a zero-sum game. It's either a perfectly
kept secret, or the secret's out there, and we're hosed. So how do we reduce the stakes? How do we make this less
severe than all or nothing? And the way you have to do it
is making secrets ephemeral. You don't want to have these
long-lived credentials that are valid forever. You want these things to be
short-lived, time-bounded, things so that, even if
they get logged to disk-- oops-- they're only
valid for so long before we've moved on to a
different set of credentials. So Vault's approach
to this is what we call dynamic secrets, right? And the idea is, as opposed
to static secrets, what we want to do is generate
a unique set of credentials on demand. So when a web server says, I
want a database credential, we generate that credential as
in point-in-time right there. We have a unique audit trail
now that ties us back and says, this web server has
this unique credential. And by the way, that's
valid only for 24 hours. So when the web server renders
a 500 diagnostic page and leaks out our database password,
we've time-bounded how long that credential's
valid before moving on to another credential. And if there is a
breach, and we know that that credential's exposed,
we can revoke it early. So what does this actually
look like from the perspective of a client? They're just doing
a read against Vault in the same way we just
saw before where they just did a read of secret/foo. And behind the scenes,
Vault is then going out and talking to the endpoint. So it's going and talking
to the database and saying, create a new dynamic credential. It's creating an audit
trail of that credential, and then it's providing that
credential back to the user. In this example,
I'm using a database because I think it's
sort of an intuitive one. But you can imagine
this exact same diagram where you replace the
database with any sort of endpoint system that
supports credentialing, right? A prime example would
be Google Cloud, itself. So I want to give my
application a credential that lets it read and write from
Blobstore, as an example. But I don't necessarily want
to give it a credential that lives forever. I want to give it a credential
that's valid for 24 hours, and I'm rotating that
credential all the time. So in that same setting, Vault
would connect to Google Cloud, generate a short-lived
credential, create the audit trail, and
provide it to the application or end user, maybe a developer's
doing some debugging, so this example holds. So when we talk about this
secret back end approach-- so far we talked about Google. We talked about using it
with a database-type system. But it's really a general idea. The general idea
is, any system that uses credentialing to manage
authentication or authorization has this similar sort
of a challenge, which is our consumers need that
credential so they can consume the end point system. But we don't want these
things to be static. And so our goal
with Vault is, this is actually a plug-in point. It's an extensibility point
where, over time, we're constantly adding new
systems and adding support for them to be consumed
in this dynamic way. So cloud providers
are one great example where they need API
tokens to interface with, and maybe are giving
those to humans, maybe are giving
those to applications, maybe your automation
systems are consuming it. How do we make sure
that we don't have a forever token in Jenkins? Databases are great
a example of systems of a record that have data. But then you get into all
sorts of other systems-- messaging queues, things
like RabbitMQ, SSH. I think we could probably--
maybe I'll spare everyone from raising hands. But I'm sure many of us
have seen the master shared PEM file with a few
hundred developers having access to the same PEM
that secures a huge fleet. And so that's an
interesting challenge where we have this shared
mechanism, this PEM file, that too many people
have access to. So the list goes on and on. Other stuff that
we don't have time to touch onto in
much more detail is classic security
bread and butter-- the three A's. As you'd expect, Vault has
many different mechanisms to authenticate against it. So whether it's an app
running in Kubernetes, and we're tying
into the platform, it's a VM just running
in Google Cloud, or it's a user who's using
LDAP to authenticate-- those many different
authentication mechanisms-- all
of that comes into, then, a central
authorization system. So there's a
central way in Vault where you authorize who
is allowed to do what. So one huge goal with
Vault is, how do we get to least privilege? You should only
be able to access exactly what you need,
not everything because you were able to log in. So Vault has a very
fine grain AuthZ system that lets you scope
down who can do what based on their identity. And then lastly
is rich auditing. So we want to have that audit
trail of who did what when. Cool. And with that, I'm going
to hand it back off to Seth who's going to talk
to us about how we actually use this and operationalize
this within Kubernetes. Thank you. [APPLAUSE] SETH VARGO: Great. Thank you, Armon. So the goal of the
next section here is to really answer
two questions. Why would I run
this on Kubernetes? Is it a good idea? Should I? And how do I do it? The first answer
is yes, so we'll move onto the second question. So what does this
actually look like? So the following diagram
here is also captured as code on GitHub,
github.com/sethvargo/vaultongke. But I want to walk through
it quickly together. And before I do,
I want to mention that this follows the
HashiCorp best practices guide for production deployments
and production hardening. This was built in collaboration
with some of the Google Security engineers as
well as some of the people on the SIC security
team on Kubernetes. So it represents what we like to
think of as the best-practices way to run Vault on GKE
or any Kubernetes cluster. So at the top level, we have
some type of load balancer. In this example,
we're using cloud load balancing because we're making
this Vault server publicly accessible. In your own
environment, you might want to use VPCs or
VPN-type networking, where it's only accessible
in certain subnets. And then we have a highly
available Vault cluster. So we might have three, five,
seven servers all running behind this load balancer. And then below that,
we have, obviously, the Kubernetes engine itself. And then we're using Google
Cloud Storage as the underlying storage back end. We also support
Google Cloud Spanner. And we are using
Google Cloud KMS to do the initial encryption
of the initial root token and the initial unsealed keys. So let's walk
through some of those in a little bit more detail. So first, we're using a
dedicated Kubernetes cluster. And there are a few
reasons for this. But we don't want Vault running
alongside the same Kubernetes cluster where your
applications are running. In fact, we might even recommend
running it in a separate Google project that only the security
teams and the networking teams-- least privilege access. And to the other clusters and
other application developers, Vault is just a thing
at an IP address. We're leveraging
Google Cloud Storage. Again, we could leverage
Google Cloud Spanner. But both of those storage back
ends support high availability. As a production
deployment, we need this to be highly available. There's no reason
why we wouldn't. So we're going to leverage
Google Cloud Storage for our high availability
and our storage back end. Another advantage there is
that we can back that data up. We can do replication. We can do customer-provided
encryption keys as well. We're going to leverage
Google Cloud KMS. Everyone taking
a picture-- nice. We're going to leverage
Google Cloud KMS because our Vault servers
are going to be automatically initialized and unsealed. So those initial unsealed keys
and that initial route token eventually need to get into
the hands of an operator or an automation
system, but we don't want them stored in plain text. So we're going to
leverage Google Cloud KMS to encrypt those values and put
them in the storage back end. Additionally, we're leveraging
our own custom certificate authority for TLS. So one of the questions
I get is, like, oh, why wouldn't you
use Let's Encrypt? Let's Encrypt is
amazing, but the problem is, I don't actually want
to trust any certificate on the internet. Instead, I want to use
mutual TLS validation, and I want to use my own
certificate authority and only trust certificates
signed by that authority, not even my system certificates. This helps secure
Vault and make sure that only people
with the certificates and the proper
certificate authority have the ability to
communicate with the server. Additionally, we're
leveraging L4 load balancing. Instead of using something
like the Global Load Balancer, the GLB, we're
actually using a regional load balancers. And the key reason
here is that an L7 load balancer is going to do
the TLS termination for us. And we don't actually want that. We want Vault to do the
TLS termination, primarily, so that Vault can handle
mutual TLS authentication. But also, we don't want
to decrypt data and then have a different re-encrypted
connection on the back end, right? Using an L4 balancer lets us
pass that through directly to Vault and lets Vault
handle its own TLS. And if Vault ever rotates
those TLS certificates that it's using, those
automatically happen. We don't have to
also rotate them in an upstream load balancer. And then lastly-- and this
is a really key point-- we want to think of Vault
as a service, right? So in my internal operations
team, or my internal security team, yes, I'm running
Vault under Kubernetes, and I'm leveraging a
lot of the functionality that Kubernetes provides. But ultimately, I'm delivering
secrets as a service or secrets management as a service. So there is Vault and an
API and an IP address. And users ultimately
shouldn't care whether Vault is running on a
VM or a container or bare metal. It doesn't really matter. So we really want to think about
this as Kubernetes is providing the building blocks
that let us run Vault as a highly-available service. And we get some
really cool benefits. One of them is you might
see that we're going to leverage a StatefulSet here. And the first thing you
might think of is, wait, this isn't a database. There's actually no state. Vault's state is
stored elsewhere. It's stored in GCS. Why would he use a StatefulSet? And there are a few
reasons why we might want to use a StatefulSet. First, it gives us
predictable naming. Our actual container names
will be like vault-0, vault-1, vault-2. This is really
helpful for debugging, audit logging, et cetera. The real reason, though, is
that the StatefulSets give us exactly-once semantics for free. So as our servers are starting
up, one will start up, it will unseal itself,
it will be ready to go. As soon as its health
check is passing, the next one will move along. So we actually avoid
any real-life [? race ?] conditions with
initialization or unsealing. The same is true for upgrading. You don't want to upgrade all
of your Vault clusters at once. You generally want to upgrade
one, and let the leadership transition to a new server,
upgrade that one, et cetera, until the entire
cluster is upgraded. This is true with any software. Using StatefulSets, a single
change to the YAML file, or whatever you're using for the
description of this deployment, automatically does
a rolling update. And if one of them
fails, it'll stop. We get a free circuit
breaker pattern. So there's a lot of
reasons why we might want to use a StatefulSet here. Additionally, we're leveraging
anti-affinity rules. So anti-affinity rules
are a Kubernetes future which let us do things
that affect or influence the pod scheduling. By default, if we did
not have this YAML here, Kubernetes would put all
three Vault containers on the first node. And that's not highly available. If that one VM dies, we lose
all of our Vault servers. So what we're doing is
leveraging anti-affinity, basically, suggesting to
Kubernetes, or in this example requiring Kubernetes,
to spread these out across all of our
available nodes or hosts. This means, if one VM
dies or two VMs die, that our Vault servers-- the actual containers
themselves-- are spread across the node pool. So this adds an additional
layer of availability for us. We're also leveraging
auto unsealing. The auto unsealing is
also an auto initializer. This allows you to spin up
a Vault cluster very quickly without operator intervention. And the keys will be
encrypted with KMS and put in the storage back end. We're also doing
some little things that make a big difference. For example, we're
giving the container the IPC_LOCK capability. By default, Vault
supports memory locking. This is something the
Vault does in its codebase. Containers don't give
you that privilege by default, so we
explicitly ask Kubernetes to give our containers the
ability to do memory locking-- again, just an added
layer of security. And then our readiness
check is basically asking Vault for its own
health, but we explicitly want to make sure that
our standbys are also added to the load balancer. Because this
readiness check here is going to determine
when the container is able to receive requests
from the load balancer, and we want to make sure that
our standby servers are also able to receive requests so
that they can forward them appropriately to the leader. Notice that we're
communicating via TLS. And then lastly, as
we saw in the diagram, we have this load balancer. Again, this might be an internal
load balancer in your own VPC. Or it might be a
public-facing load balancer. I don't recommend that, but
there's no reason it can't be. So with that, let's
jump over to demo one-- so if we can switch
over to the laptop. All right. Here we go. Did everyone pray
to the demo gods? Here we go. All right. So the first demo
here, I'm just going to show you what it's like
to run Vault as a service on Kubernetes, basically, using
those exact YAML files that I just showed you. First, I want to show you, I
have a Kubernetes cluster here. It's running on GKE. So I'll go to Kubernetes
Engine, and we'll take a look at the workloads here. And there's nothing, right? We have no workloads here. We haven't deployed Vault yet. So what I'm going
to do is, over here, I'll just take a look
at my vault.yaml. And you can see that this is
very, very similar to what I just showed you on the slides. There are some additional
pieces of information here, like, I'm deploying
a replica set of three. So I'm going to have
three Vault servers. Again, there's
that anti-affinity that we saw before. And all of this is open
source, by the way. So don't feel like you have to
capture all of this right now. We have our first
container, which is our init container
that's going to do the Vault initialization. And then we have
some configuration for that container. And then we have the
actual Vault server itself. We're pulling that directly
from the Docker hub. Again, there's that
IPC_LOCK capability. And we're opening
some container ports that will be available
for requests-- some basic memory
and CPU requests, and then, again,
some configuration. And here, you can see
where we're actually supplying Vault its
startup configuration via the magic VAULT_LOCAL_CONFIG
environment variable. So we don't have to worry about
writing out a file to disk. We do this all in the
environment variable. So with that, let me go
ahead and apply this. I'm terrible at typing,
so I wrote fancy scripts for you all. So what this is
going to do is it's going to just grab that
Kubernetes end point, and it's going to apply
kubeclt apply-f that particular StatefulSet. So what we should see
here in a second-- and this is where everyone's
prayers would be helpful-- OK, there we go. So these pods are
actually starting up. And you'll notice that 1 of
3 of them are starting up. And that's because we're
using the StatefulSet, which is doing that one at a time,
exactly-once semantics. So vault-0 is running. It has some
warnings, but I think that's just because
it's booting up. And if we go back, we'll
start to see, hopefully-- OK, vault-1 is starting as well. And then once we get to
vault-3, this entire replica set will be happy. And at that point, we
will have a vault cluster. Because what's actually
happening, under the hood, is we're not just running
that Vault container, we're also running that
Vault init container. And because this is a
fresh, new container, it's actually connecting
to the storage back end, doing the
initialization, putting those unsealed keys encrypted
via KMS into that storage back end, as well as the
initial route token. And then the other
servers, when they start up, they look,
and they say, oh, I'm already initialized. They grab those encryption
keys and do automatic unsealing for themselves. If we were to
scale-up or scale-down the cluster, because the
initialization has already happened, the initializer
will just unseal them. It won't actually do the
initialization anymore. Let's go ahead and refresh this
and check where our status is. I see a green check
mark, which is great. I'm going to go ahead and jump
over to Google Cloud Storage here, just real quick
to show you all. So this is the storage
bucket that I'm using. And in here, you'll
see the folders here-- core and sys, which
are created by Vault. But then the root token
and the unsealed keys are both encrypted
via Google Cloud KMS. And then they're stored in this
Google Cloud Storage bucket. So when I want to interact
with Vault, as an operator, I need to download
those via like gsutil or via the Google
Cloud API, decrypt them using Google Cloud KMS
and the proper IAM permissions, and then I can interact
with Vault directly. So let me go ahead and
switch back to Kubernetes, and I'll show you what
all of that looks like. So I'm going to jump over here. I have another
magical shell script, which I will show everyone. So this is my
magical shell script where I super can't type. But basically, all of this does
is it grabs the CA certificate. Again, we're using a custom CA-- so putting the output of
that, the address to my Vault cluster. And I provisioned this Vault
cluster with Terraform, so I'm delegating
to Terraform there-- and then the token
decrypt command, which is just a really
long gsutl command with some basic Z4
and some G Cloud KMS. So what I'm going to go
ahead and do is run this. And again, this is
all open source. Oh, look. See, I left myself
this note because I knew I was going to mess it up. This is when you know
you've done too many demos. So this takes a little
bit because it's actually making about 5 or
10 API calls, but great. Now, what I can do is
I can actually just run env grep vault. We
see that we have our VAULT_TOKEN, our VAULT_ADDR,
and our VAULT_CACERT. These are magic environment
variables the Vault reads. I can also supply them
via the CLI, but I'm lazy. So now what I can
do is, hopefully, I can run Vault status. And look at that. We now have a Vault
cluster up and running-- five key shares, a
threshold of three. It's in high-availability mode,
and this particular node I hit is in standby. So if we look at that
second to last line there, we'll see that this particular
node that we hit is in standby. If I run Vault status again,
now I hit the active node, coincidentally. Because we're doing round-robin
load balancing on that load balancer there. Anytime I make a
request to a standby, that standby will automatically
forward the request to the active leader. So I don't have to know
where the leader is. I just know the load
balancer address. So this is how you run Vault
as a service on Kubernetes. If I kill one of these pods,
it will automatically restart. It'll maintain a
quorum of three. When it comes back online, it
will be automatically unsealed. And until that point, it won't
be added to the load balancer as well, which is why that
health check is so important. So with that, can we
jump back to the slides? Awesome. So that concludes
the first demo. You can clap. [APPLAUSE] So now we have to
forget everything. Forget everything you
just talked about. Because the next section is,
now that I have Vault running, how do I connect to
it from Kubernetes? But I want you to forget that
Vault is running in Kubernetes. Think, it's running in
some third-party service like Heroku, or it's
running on VM somewhere. Forget that Vault is
running in Kubernetes. Instead, Vault is just a thing
available at an IP address. Everything we're
about to do can work with any Kubernetes cluster. We're going to use
GKE, but it could work with your on-premise
Kubernetes cluster, et cetera. So again, forget that Vault is
running on GKE or Kubernetes. It's just a thing
at an IP address. Because the next
thing we're going to do is create another
Kubernetes cluster. All right. Clicky. All right. So very quickly, we have to
talk about Vault authentication methods. So Armon mentioned this very
briefly in the beginning. Authentication in
Vault is the process by which users or
machines supply information to Vault. Vault
verifies that information with, either internal data or
some external third party, and then, assuming
that it is valid, converts that into a token with
policy authorization attached to it. Here's an example of what that
looks like in a graphic form. We have the Vault server
in the middle, which would receive a
request from a user or a machine or an application. It would then take
that information to this third-party system. Assuming that's successful,
it would go to its own policy engine and map that
to a Vault policy, and then return the token,
which is kind of like a session ID or a session token to the
user, which would then be used for all future requests. It's like logging
into a website. You know, sometimes you log in
with a username and a password, and your browser
sets a session ID. But sometimes you can
also log in with Twitter or log in with GitHub. Vault is conceptually
similar to that. So what does that look
like with Kubernetes? Well, it's actually
very straightforward. We just replace that third-party
system with a token reviewer API from Kubernetes. So a user, or in
this case a service account representing
a user or a pod, makes a request to the HashiCorp
Vault server with their service account token, their JWT token. That JWT token is given
to the TokenReviewer API. The TokenReviewer API verifies
that JWT token's validity, and time stamp, et cetera. And assuming that's successful,
it goes back to Vault and says, yeah, this looks great. Vault then goes to its
own internal policy engine and says, oh, OK. Well, the things in this
particular namespace get this particular
policy attached to them. Here's a Vault token with
that policy attached. And this is how the
Kubernetes auth method works at a very, very high level. There are technical
details here, which we're not
going to get into. But this is, at a very high
level, how this functions. So to do this, you could write
your own little BASH script. Or I've actually open-sourced
the Vault Kubernetes authenticator, which does
this whole process for you. It will grab your
service account from var/run,
Kubernetes, secrets, the service account JWT token. It will post that
up to the Vault server which is configured
via an environment variable. It will get the
token back, and it'll put it in an in-memory
volume mount for you. So then, all your applications
and services have to do is extract that token from
that in-memory volume mount. So you don't have
to actually think about Vault authentication
in Kubernetes anymore. You could just
leverage this and know that the token will be there,
assuming authentication was successful. It runs as an init container,
not a sidecar container. So it must succeed in order for
the other containers to run. Then we leverage tools
like Consul Template. So despite its
name, you don't need Consul to use Consul Template. But Consul Template
is a tool that is designed to provide a
configuration language by which we extract secrets and
credentials from Vault and present them on-disk or on
a temp disk or an in-memory disk to an application or service
without the application or service knowing that it's
communicating with Vault. Many of you have brownfield
applications or, even, greenfield applications
that you likely don't want to directly integrate
with Vault. You might not want to call the Vault API
directly using the Golang library or the Ruby library. Instead, you might
want to just continue to leverage a file
system or environment variables for credentials. And we like that abstraction
because then you're not tightly coupling
your applications to a secret manager. Instead, you're saying,
my config is on disk, or my config is from an
environment variable. What Consul Template
allows us to do is to provide that
same experience, but under the hood, as
a sidecar application, it is actually
managing the connection to Vault, the renewal and
the acquisition of all of those secrets. And as we'll talk
about in a minute, what happens when those secrets
are revoked or are rotated? So I know what you're thinking. What happens, though, if
my credentials change? So I have this
sidecar application that wrote this file out. And my application is
reading those credentials from that file. But how does this
Consul Template thing actually inform my application
that my credentials have changed? So let's think of a
real-world scenario. I have a database credential. Vault tells me that
database credential is valid for 24 hours. At the end of 24
hours, Vault is going to revoke that credential,
if it's not revoked earlier by an operator. At that point, my application,
or Consul Template, has to go back to Vault and
say, OK, I'm still authorized. I would like another
database credential. And it will be different. It will be a different username
and a different password. So I'll write that
out to my config file, but my application is probably
already loaded its config. Most web applications don't go
to disk and read their config every user request. They load that into memory,
and they kind of cache it until they're restarted or
received some sort of signal. So how can Consul Template
signal my application to reread its credentials,
where, in a Kubernetes world, where all those
containers are isolated? Well, an interesting feature,
and an alpha feature right now in Kubernetes is this idea
of shared process namespaces. So shared process namespaces are
the solution to this challenge. And again, this is
an alpha feature. So it's available in GKE alpha. And if you're running your
own Kubernetes clusters, you'll have to enable it with
Kubeadm or a similar tool. But this shared
process namespace allows our containers to
communicate with one another. So instead of having every
container be PID 0 or PID 1, they're actually a
root container and then these subcontainers
with the ability to communicate with one another. So if we give our
Consul Template process the SYS_PTRACE sys call, the
security context capability, we can actually leverage the
standard Unix tools like pidof and kill to signal applications. And then, all we have to
do is provide a command to Consul Template. And it says, hey, when
this file changes, or when this
environment is updated, or this secret is revoked,
just send, in this example HUP to the pid of my app. And we can leverage the
built-in pidof tool for that. Or perhaps it's
something more complex. We might have some
50-line bash script that we have to run because
of some arcane Java-based application that
is very finicky. We capture that as code,
and Consul Template executes that for us. So this can either be
as simple as a signal, or it can be as complex
as a very systematic and procedural restart. So with that, let's go ahead
and jump over to the laptop, and we'll take a look
at what demo two is. So what I'm going to do here is,
in the Google Project Switcher, I'm actually going to switch
to a different project. So I have another project
here called My App. So I'm in a completely
different project now. Remember, one of
our best practices we recommended earlier was
that we have a dedicated project and a dedicated
Kubernetes cluster for Vault. And then, we're only thinking
about Vault as a service right now. It's just a thing
at an IP address. So I have this other
Kubernetes cluster where I run all my applications
and services for production. All my web apps, all
my stateless services, all my stateful
services, they're all in this My Apps
Kubernetes cluster. Right now, I don't
have any apps, but we're going to
create a few in a second. All right. So I have a few scripts here. The first is we're going to set
up the Kubernetes auth method. What the Kubernetes
auth method is going to let us do is, like
we described in that diagram, it's going to configure Vault to
talk to the Kubernetes cluster with the right
certificate authority, so that we can verify via the
Token Reviewer API, the JWT tokens. So let's go ahead and run that. So it creates a CRB
and a few other things. And then we actually configure
the Kubernetes auth method, and we create a new
role for authenticating to that configuration. So now, at a very high
level, what this means is, when I have a
pod in my cluster, and it goes to Vault
with its service account, that service account
will be authenticated to get a Vault token back
with a predefined policy that I've configured. So this is configuration that
you could capture as code, or you could run it
manually via the API. But then next,
what we want to do is we want to configure
our GCP secrets. So the use case here is that-- Armon's example was
database credentials. You might be Postgres
or MySQL, et cetera. One of the other secrets
engines that we have for GCP is the IAM GCP secrets engine. This lets you programmatically
generate both service accounts and service account
keys for GCP. So instead of going
into the web UI, and clicking, or using
the API directly, you can programmatically
generate on-the-fly tokens that are very short lived
for interacting with the GCP API, the same way that you might
generate like an AWS access key. But we can do that
with the GCP API. So what I've done
here is I've just configured Vault, very briefly,
to interact with those systems. So the last thing
we need to do then is actually apply
our application. So let me show you what
our application looks like. So I'm doing a
bunch of stuff here, mostly is just because
I'm really bad at typing. But I'm grabbing the
credentials from our cluster, and then I'm
setting the context, and then I'm applying
this Kubernetes YAML file. So I'll take a look at
that YAML file here. So there are a few things
that I want to point out here. The first is that we're
running a pod now. We're now no longer
running a StatefulSet. This is just like
a straight-up pod. It could be replicas, but
there's only one at this point. We're leveraging that shared
process namespace again. This is important so that we
can signal our applications so Consul Template can
talk to our other process. We have a few volume mounts. We have our secrets. This is where Consul
Template's actually going to write its
configurations because there are going to be secrets there. We have our own
TLS configuration. This is how our pod talks
to Vault because, again, we need that certificate authority,
and client-side TLS cert, and then the Vault
token, which is an in-memory volume mount where
the token will be written. So that never gets
persisted to disk. We have our init container here,
which is, again, just pulling from the Docker Hub. And we have a few
volume mounts there. We set some
environment variables, like where is the Vault server,
where is its CA certificate. And that's going to run at boot. And assuming that's
successful, it'll put our Vault token at a
well-known path on disk/home Vault. And at that point,
all of our other containers-- in this example,
Consul Template-- can then pull that token from
the in-memory file system and interact with it. So here we have Consul Template. We've given it the SYS_PTRACE
capability so that it can signal our other application. And if we scroll down
here a little bit, again, same volume mount,
same configuration. And then here's our Consul
Template configuration. And at the very end here,
you'll notice that we have our arbitrary command. Our arbitrary command
is just to send kill-HUP to the pit of our
Vault demo app. And then, as you might
suspect, the last thing in our configuration here is
our Vault demo app itself, which is just getting pulled
directly from the Docker Hub. And actually, all it
does is log its own file, so it's not very exciting. But it's really good for
demonstration purposes. So let me go ahead
and apply this now. So now, if we go back here, we
should see this workload start to provision. And you'll see that the
pod is initializing. And if I'm fast enough-- I wasn't fast enough. If you're fast
enough, you can see that it's waiting for the
init container to run. But if we take a look
at this init container-- and I'm using
Stackdriver Logging here, so you'll actually
see the logs directly in the Google Console UI here. We see that, a few seconds
ago, that the init container successfully stored
the Vault token. So it did that whole walkthrough
of going to the Vault server, presenting its JWT
token, interacting with the Vault cluster. And then what we presented
to Consul Template was just a Vault token
at a well-known path. Consul Template then
grabbed that Vault token and used that token to
interact with the Vault API to grab our service account key. So now, when I go back to my
pod and refresh this here, we can see that both
Consul Template and My App are running. And if we take a look at
the logs for all of these, you'll notice that Consul
Template sent a signal to My App causing it to reload. So at boot, Consul
Template detected a change in configuration. There was no config, and
then it went to some config. So it sent the HUP signal
to the Vault demo app. The Vault demo app, or My App
in this log, got that signal and printed out a log
line that it reloaded. What you see here is a
soon-to-be-revoked access credential for interacting with
this particular GCP project. This is an actual IAM access
token for interacting with GCP. I'm aware of this. I will revoke it, but it's cool. So I'm showing it to all of you. And what's happening
is, at some interval, this token just
gets printed out, basically, every 30 seconds. However, at the end of about
five minutes, what will happen is Vault will revoke that token. So by default-- like
GCP IAM credentials live forever until a human
comes in and deletes them. Vault actually
enforces a lifetime, and it makes the necessary
API calls to Google Cloud to revoke those credentials
when time permits. So, unfortunately, we're
running short on time, so I can't actually
wait and show you that. But if we can switch back
to the slides very quickly. So the last section now I'd like
to invite Armon back on stage to share a little bit
more about the Vault roadmap and the future. So, Armon? [APPLAUSE] ARMON DADGAR: Thank you. Awesome. We are definitely
running short on time. OK, we're going to
make this light speed. So thank you, Seth, for walking
us through a little bit more hands-on than I had a chance
to go to in terms of how we actually work with it. Some of the stuff I
wanted to talk about-- so for folks that are maybe
familiar with Vault, or have already
used it, I wanted to give a little
bit of a sneak peek in terms of where Vault's
going and a little bit of our thinking in
terms of directionality. I think one of the things
that's become clear to us is that, when we talk about this
space, secret management, tools like Vault, integrating it
with these platforms is, it's complicated. And I think you kind of
saw a snapshot of that is, it's not exactly simple and
obvious how all of these things kind of come together because
it's a hard problem we're trying to solve. We're trying to solve the
problems of application identity, brokering between
many different systems that don't necessarily
understand each other-- like Postgres has really no
idea what Google Cloud's IAM model is and vise versa. So what we're trying to do is
solve this challenge of each of these systems, implement
different mechanisms of access control, and we're sort
of brokering between them. So it's a complex problem. And so how do we start making
that a little bit simpler? So one of the things
we're starting to work on within Vault itself is-- I jokingly refer to
as sort of Clippy for Vault. As you can see
a little pop-in wizard in the corner here,
which says, "It looks like you're trying to
configure Vault. Can I help you with that?" But in more seriousness, how
do we help guide users through, hey, these are
the different kind of core workflows
in terms of how do you configure Vault
to understand Kubernetes as an authentication back end? How do you configure Vault to
integrate it with Postgres? How does the role-mapping
work between this. And so it's meant to be
an unobtrusive wizard-like capability for
different workflows that you can enable as
you hit them to say, hey, please walk me through
how this is supposed to work and have it be a little bit more
interactive as you're getting started with the system. Another common challenge
we see with Vault is, how do I go from-- I think Seth did a great
job laying out, what's a best practice deployment
guide look like in terms of isolating it with a separate
cluster and treating it as just opaquely living behind
a load balancer and building HA. And so an interesting
pattern is, how do we go from providing that
service, Vault as a service, for a single application
owner or a single group to, all of a sudden, in
many application owners, many groups? We want Vault as a service
to be highly multitenant between many different potential
groups or lines of business. And so one approach
to this, and the kind of historical
approach, is you lean on Vault's very fine-grained
capability and an authorization system. So you can segment out
different parts of the system. Maybe you say, [? eng ?] star
is managed by one set of people and [? marketing ?] star is
managed by a different set of people. And we sort of segment using
Vault's hierarchical folder structure. Another capability, a different
way of thinking about it, is full namespacing. And this is what's coming on
our roadmap is the ability to carve out entire
namespaces within Vault and say, you know what? I don't want to
think about managing permissions of these folders. I want to carve out a
whole namespace and say, this belongs to engineering,
this belongs to marketing, this belongs to another team. And those teams can then sub
administer these namespaces independently. So looking at, as we go and
think about Vault as a service, how do we scale up the
number of consumers, the different folks who can be
onbaorded onto this service? Lastly is some of the challenge
around integrating with Vault, as Seth highlighted is there is
this tension between what we'd like to be able
to do is maintain operational flexibility
and not tightly couple our applications. We don't want all of our
applications to embed vault SDK and directly talk to the API
and have to be sort of highly Vault-aware. That sort of limits our
operational flexibility. So instead, we'd like
to decouple that and use tools like Consul Template
and scripts like Seth's auto initializater to thread
through the right credentials so that our app boots,
reads its secrets, and it's blissfully unaware. A really common
challenge, though, is how do we do that
authentication before we can actually read secrets? And so one of the things we're
starting to work on-- and it actually just shipped
this morning in beta-- is the Vault Agent. And so this is an
optional agent you can choose to run alongside
Vault Server, which you have to run. And what it does is
manage some of that flow for you, the flow
of fetching the JWT, authenticating against Vault,
periodically reauthenticating, integrating with tools
like Consul Template to be able to generate and
put configuration on disk in a more
application-friendly format. So these are some of the goals
of the Agent is making it easier to authenticate, easier
to manage token renewal, easier to cache some
of this data locally, and making those integrations
a little more out-of-the-box. So this is something
that just shipped today. So if this is something
that you find useful, please feel free
to play with it. So with that, thank you, guys
so much for sitting through with us and joining us. And thank you to Seth as
well for helping with this. [MUSIC PLAYING]
