[MUSIC PLAYING] DAVE STANKE: All right,
so my name is Dave Stanke. I'm part of the Developer
Relations team here at Google. And with me is
Christopher Sanson. He's the product
manager for Cloud Build. He'll be up a little later. And that character
up in the top right, that's Billy the
Builder Bee, and she'll be helping us out later on. So we're shifting left. What does that mean? It means to incorporate critical
processes early in the software development cycle. What does that mean? Well, we can usually model a
software development process as a left to right
flow, where we go from code, through
some transformations, see if it's any good,
ship it, and then operate, monitor on it, as it's live. And we model this as
a left to right flow. Shifting left means moving the
important steps of validation and quality control
earlier in the process, closer to the left. But of course,
software development doesn't happen in
a single stream. We usually have
lots of these flows. And in a traditional process,
every developer or team progresses independently. They're focused on writing
more and more code, until at the last minute,
everything collides. And all at once,
all of these bits are meeting for the first time. And we're hoping for a quick,
thumbs up, let's ship it. But when you have a
whole bunch of code, it's strangers to each other. And you shove them
all in a room at once, they might not get along. And so it's more
likely that we're going to have merge conflicts. We're going to have
terrible security holes. We're going to have
incompatibilities between some systems. And this is why in a traditional
waterfall-style development process, you often have
huge, lengthy merge times. You often have long lead times. And you often have
terrible weeks before and after
a major release. So we shift left. We move processes earlier
to get earlier feedback on small checks
all along the way. So if you take two small pieces
and you validate each of them independently, and
then when they combine, the only thing we
actually have to test then is what arises between the
connections between them. So what are we shifting left? Well, you can shift
left lots of things. First off, you can
shift your merges left. And this is
continuous integration in the original meaning-- that instead of having
long-lived branches that hang out on a developer's
machine for a long time, we're going to merge
at least once a day, preferably more often. And each one of
those merges is then small, relatively pain-free. And we don't have "the
merge" that everyone dreads. We can also shift
left on security. So a lot of times,
traditionally, we put together a big
software release. And then just before we ship,
we go to the security people and we're like,
it's good, right? Give us a thumbs up. And they say, slow down. Well, we can do a lot better
if we do security validation earlier and oftener. So within code,
you can do things like automated fuzz
testing to make sure that your code actually
has some protection. You can test
individual components, look at open source
packages you might be using, using
vulnerability analysis, and generally have affirmation
that your code is pretty secure all along the way. And at the end, you're
only validating, again, that very outer
perimeter of the code. And then you can
shift left on testing. And this is what we're
going to really dig into-- testing early and often. And why do we want
to test early? To catch bugs early. I'd bet a lot of folks
here are familiar with the research that's
been done by NIST and IBM talking about how finding
a bug in development costs way less time and energy
than finding one in production. The numbers I've heard
are anywhere from 6x, to 30x, to 100x. But certainly, you'd
rather catch the bug early. Why? Why is it so much cheaper? Well, I think there's
a couple of reasons. One, if you catch it
early, it hasn't had time to have downstream effects. The other system components
aren't aware of that bug. They're not working around
that bug, as they sometimes do. And two, if you catch a bug
while a developer's still working on that process,
it's still in their mind. There's still in that
flow, and so they don't have to rehydrate
and reconstruct, what was I thinking when I wrote
that code with the bug in it? So for these reasons,
it's really great to fix our bugs early
and be testing early. Oh, and one quick
bit of science here. From the State of
DevOps Report-- and actually, I
believe there's a card on every one of your
seats encouraging you to participate-- the
State of DevOps Report does analysis finding
out what practices are done by teams that have
the most software quality. And they find that continuous
testing predicts continuous delivery, which predicts
delivery performance-- how often we ship, and how
fast we get from code to live-- and availability,
how much of the time our services are available. All right, so let's look at
testing and how we can shift left on it. First of all, some definitions. So we often talk about testing
as the testing hierarchy. We start at the bottom
with unit tests. And unit tests, of course,
are the foundation. Unit tests are where we
test small functionality of our code, kind of within
the frameworks of our code. And honestly, these are
a mostly solved problem-- maybe. But most modern frameworks
have unit testing built in. It generally can be run
pretty quickly as part of your build and CI process. Integration testing is when
we test multiple components together and when we make sure
that, when these things are actually running, like
in production, that when they work together, they don't
all of a sudden fall apart. This one worked. That one worked. But the application doesn't work
until you have both of them. And on top that we've
end-to-end testing, which means usually running
a multi-step process against a composed application. There's kind of a fuzzy line
between integration testing and end-to-end testing. And the terms are sometimes
used interchangeably. Now, this diagram might look
like a tall, intimidating icy mountain, but we're
going to show you that it's not that hard to scale
to the pinnacle of testing. Today's focus is on
integration testing, but everything we're looking
at can be used for E2E testing, as well. So our goals for
integration testing-- we want high fidelity, high
isolation, high scalability, appropriate ephemerality. And we always like
speed and low cost. We're not going to
get all the way, but these are an
aspiration to shoot for. I'm going to show you some
ways that we can get close. So first of all, fidelity. Now, when we do
integration testing, we start by creating
a system under test. That's the environment that
we're going to test against. We're not going to test
against prod, right? And so we create a
simulation, effectively. And our goal here is fidelity. How similar is that system under
test to the production system? The closer we get,
the more assured we are that we're going to have
tests that reflect what reality is going to be like, what our
users are going to experience. And we're generally not
going be able to create a full production
environment for each test, although we'll
challenge that later on. So we're going to run
against a scaled-down version of that full ecosystem
and aim to make it as predictive as possible. Isolation-- this means that
each test is running completely separate from the others. They're not going to stomp
on each other to compete for processes, CPU, memory,
disk space, whatever. And they also don't connect to
the same external resources. If you have a shared
database or a prod API, even if it's read-only,
it's entirely possible for a runaway
read to block everyone else from using it. Isolation also has
to do with security, so we want to configure
all of the resources we use within tests to not
have public visibility. The last thing we want is for
someone to corrupt our tests or somehow give us bad data that
then sneaks into prod somehow, or run a bitcoin miner on
our testing infrastructure. And then scalability--
of course, we want big. How many tests can
we run at a time? Ideally, every developer can
kick off all of their tests at the same time. We don't want developers waiting
while their tests are queued up behind their colleagues. Remember that flow we discussed? Keep the developers going. And this means we want to
be able to scale the test infrastructure elastically and
parallelize all of our tests. So that, of course, is a great
fit for cloud infrastructure. And then ephemerality,
which means, how long does that
test infrastructure hang around after we're
done doing our tests? Ideally, it lasts just
until the test is over. And then it's gone, right? We don't want to pay for it. And then we optimize the
ephemeral nature of the test. But the thing is, if the test
gets torn down immediately, then we don't have access
to go debug what happened inside that test environment. Now of course,
best practices, you should not be SSH-ing
into your prod machines. But for a test machine, you can
learn a lot from a failed test by going in and saying,
well, what's going on? What process is there? What actual data got
written into that file? Oh, it's not what I expected. So the best of
ephemerality means that the system lasts
exactly as long as I need it and not any longer. And of course, speed is good. We like fast. This is important
both so that we can keep our
developers productive and also so we can increase
our mean time to deploy-- decrease, rather. Fast is better than slow. And finally, affordability--
developer time is expensive. And generally, it's a good use
of money to give infrastructure to your developers
for their tests, but it's always going
to be a trade-off. And you certainly don't want
to pay more for your testing than it's worth for the sake
of your production environment. All right, now I'm going
to kick off a quick demo. There we go. All right, so what
I'm going to do here is I'm going to start off
a series of four tests. And I'm not going to
explain them just yet, I'm just going to
get them going. But I'm in my
editor here, and I'm running this as a
GCloud submit command. Of course, you can also run
this from a trigger, as well. So we'll kick that off. And we will head to our UI,
and we'll be able to see. This is the Cloud
console, and we're going to look at the progress. And we're just
getting started on it. And back to the slides, please. And now, Christopher's
going to come up, talk to us a little about what
testing is really like and how close we
get to those goals. CHRISTOPHER SANSON: All right. Thanks, Dave. Hey, everyone. So as Dave mentioned,
my name is Christopher. I'm a product manager at Google
working on Cloud Build, which gives you a bit of a hint of
what I might be talking about. But before we get into it, one
thing we hear from customers all the time is, you come
to an event like this, you read blogs, you read
up on the best practices, and it's really inspiring. You hear about things like shift
left, things like DevOps, SRE, infrastructure as code. But when we talk to customers
in sort of back and reality is that, look, this
is great, but this is kind of like flying cars. Like we get it, but it's so
far from where we are today and our day-to-day reality. We need help both with
tooling and operations and processes to help us
get to where we want to be. So I want to take a quick minute
to look at some test patterns and test strategies that
we see in the real world and what's happening when
things like shift left, I think, are reasonably accepted
as best practices. So I think the first test
that anyone ever runs is the build itself. Every application starts
with compiling code, running a build. And you iterate, and you test
your code, and you hack on it, and you debug. And then eventually,
finally, the build is green. And so that's, I
think, often as far as a lot of different
applications ever go. If the build is green,
my app is green, right? But obviously in
modern architectures, with things like microservices
and external dependencies, just because your
build is green really gives you no indication that
your ultimate application is going to run correctly. There's a lot of
external dependencies that can change out
from underneath you while you're working. Other APIs can shift. And so you really need to
actually test and validate that the way that
your application works isn't just complete in and of
itself, but actually in the way that it talks to other services. So the next thing we
see a lot of people do is, OK, the build
is green, and they start adding quality gates. So these are static
analysis tools, either commercial, like
Codecov or Coverty, as well as open source things,
like JSHint and JSLint, things like that. And quality gates
are really great. They are very fast. They're very cheap,
and they give you a really good indication
of code quality. They could do everything from
validating against your style guide, linting, giving you a
sense of how much code coverage you have. The challenge though, is-- for anyone that's looked ahead,
if you look at the unit test that we wrote, that
unit test isn't really going to help you very much. All it's really going to do is
just get your code coverage up to 80% so you can be allowed
to like merge your commit. So what we find
is, quality gates are really helpful and really
powerful for indicating that your code is
probably going to work, but it doesn't
actually do anything to verify that your
code will work. So after you've
had quality gates-- this is my personal favorite
test strategy and one I've often employed. I like to call them
Humpty-Dumpty tests. This is where you don't
actually write any tests till after the failure occurs. Something will go down in
prod-- you'll have an outage. Your users will talk to you
and say, hey, this is down. You figure out the recalls,
you submit a quick fix, and then you go back
and write test coverage. Obviously, this is a big
reality of software development. You're not going to
catch everything. But I think it's sort of
obvious you don't really want your users to be your QA. You want to test and
capture these things before they reach users. By the way, I wanted
to give a shout-out to our Illustrator, Tracy. This is maybe my
favorite illustration that I've ever had
at a talk I've given. We were like, well, the
idea is, it's Humpty Dumpty. It breaks. And this is what she came up
with, so shout-out to Tracy. And the last thing is classic,
almost cliched at this point. It's sort of, well, it
worked on my machine. So we ran a survey on Twitter. We asked people in
microservice architectures, where's the first point in
time that you actually test your app, versus the other
services it interacts with? And this number was actually
a lot higher than we thought. So 43% of developers
were actually testing it in their local
machine before pushing to prod. So this is actually
encouraging, but this often leads to the kind of,
it worked on my machine, the classic failure
mode where you throw it over the wall to the
deployment or ops team. And then there's
this back and forth of blame game of like,
well, it worked for me, but it's not working
for me, et cetera. This happens because
they're different versions, there's different
OSes, there's different file path dependencies. Just because you
run a test locally doesn't really give
you a ton of confidence that it's going to run
either in your CI system or in production. So why are these some of
the patterns that we see? And we're not trying to present
these as anti-patterns, right? Like, if something
goes down, you should write a test to
cover it for the next time. You should do quality gates. You should be testing locally. But I think what we
found is that, when it gets to things like
integration testing, it's just not that easy. It's not that easy to do. There's a lot of friction. Two of some of the
primary reasons, I think, just as a
developer myself, is that, the last thing you
want to do as a developer is to lose flow, is to
get out of the context and have to wait
for a long build. And if you are using
a hosted CI service that you pay per concurrency,
or if you're self-hosting your infrastructure and you're
responsible for scaling it up and down, it can lead to long
build times, long queue times. So as a developer, you
find your own workarounds. You test things locally. You write fewer tests. You test later on
in the process. So this is where
some of the tooling can really empower developers
to lower the bar of entry for testing things
earlier on in the process. And the other aspect is
just environment management. So it's simply just
somewhat difficult, often, in different situations, to
provision infrastructure, manage that infrastructure,
spin up databases and spin them down, things like that. So which leads us
to Cloud Build. So Cloud Build, if you've
been to the other talks, you may have heard of. If you're a GCB customer,
you may be using currently. But if you're not
familiar with it, Cloud Build is a
hosted CI/CD service on GCP for building, testing,
and deploying applications. It's generally available. It launched about two
years ago, originally called Container Builder. Last year at Next, we
announced it as Cloud Build, as it covered a broader
spectrum of CI/CD use cases. And I think one of
the biggest things that we hear people
really like about it is that it's serverless,
for whatever that word may mean to you. It's a fully managed
infrastructure. And you pay for what you use,
so you pay per build minute. So if you have a huge
CI infrastructure that's idle most of the
time, you don't really have to worry about that. And the last thing-- one of the
announcements this week is, Cloud Build
is adding support for hybrid and
multi-cloud workloads. So it's currently in Alpha. It's a new feature
called, Custom Workers. But it's going to let you
use Cloud Build with a VPC to connect to your
on-prem private networks. So if you have a
self-hosted repository or you're deploying to a
runtime like GKE on-prem, Cloud Build is going
to be able to work with managed infrastructure
with services and resources behind your firewall. So let's take a
look real quick-- so how Cloud Build
works, there is a build config file
that's in YAML, of course. And in that, you define what you
want Cloud Build to do for you. So you define a
series of build steps. And in that config,
you could do things like define your dependency
graph, environment variables. It integrates with
KMS, so you can do secret management,
as well as the artifacts that you want to build. So what this looks like
in your pipeline is, you have often source
code in your repository. You can set up
automated build triggers so that when you commit to your
repo based on branch, or tag, or pull request, it will
kick off a Cloud Build. And in that build, you'll
run a series of build steps. And so build steps are like
Jenkins stages, or GitHub actions, or any
sort of CI systems concept of jobs or tasks. And each build step is a
container in the Cloud Build world. So anything that
can be containerized can be run as a build step. And at the end of your
build, there's some outputs. There's a status,
usually pass/fail, any artifacts that
were created, and then any deployments that were
made, if you're using it for continuous delivery. So what are some
common build steps? We have a whole bunch of
supported builder images out-of-the-box, things like
Docker Build, Maven Build. There's a GCloud one. So any command with the Cloud
SDK you can run locally, you can run in a hosted cloud
environment with Cloud Build. And the last one is you could
bring your own container. So as I mentioned,
we have a bunch of predefined ones for
the most popular tooling. But if you have custom
tooling, custom versions you want to use, any
sort of arbitrary scripts you want to run, if
you could containerize that, then Cloud Build
can execute it for you in a hosted environment. So if you start to think
about the possibilities there, what that means is people
are often like, well, is Cloud Build just
for building things? And answer's, well, sort of
yes, but it could do a lot more. It could be used
for anything that can run inside of a container. So in the context of
integration testing-- which Dave is going to
show us a little bit more tactically how this
works in practice-- you could use some
of the common tools. So if you're using
Docker Compose, you can run a
Docker Compose step, spin up services
in the background, like databases, run
tests against that, all during your CI process. It has support for things
like terraform and kubectl. And then you can run
arbitrary scripts, as well. So if you have
existing tests that are shell scripts or Python
scripts that you want to run, you can just simply wrap
those up in a Docker file and run that, as well. So just to wrap up
before I hand it off-- so why Cloud Build? I think first and
foremost, it's serverless. It handles a lot of the
boring stuff for you. It scales up and down. You pay for what you use. And it's isolated. So every build step runs
inside of a container, and every build
runs inside of a VM. So a VM is your isolation
boundary, and it's ephemeral. So we have workers ready
to go to run your jobs. And at the end of a build,
we'll spin down the VM. So anything that you may have
spun up during the time sort of goes away. You don't really have
to worry about managing state through persistent
environments, things like that. And the last is the flexibility. So we give you a lot of the
out-of-the-box tools that are the most common use cases. But with things like
tests, a lot of teams have their own processes,
have their own tooling that they wrote around this. And really, Cloud
Build allows you to run anything that, more
or less, you could run. And so with that I'm going
to hand it back to Dave to show us this in
action a little bit more. DAVE STANKE: Thank
you, Christopher. All right, so let's
take a look at how we can use Cloud
Build to actually do some of this cool
integration testing stuff. So I'm going to show you a
demo that includes this sample application. This is a simplistic version
of a microservice application-- just two tiers. We've got a web
app that's running Node.js and a DB that's MySQL. But of course, as engineers,
we know that the only numbers are 0, 1, and
infinity, which means this has an infinite
number of microservices. You can definitely scale
up these techniques. And this just renders a
cloud cookie shop app. And it runs inside Kubernetes. You can use these
techniques for any kind of integration testing. But here, we're focused
on a microservice app with the deployed
target Kubernetes. Integration test
flow-- first, we're going to build our containers. We have the web and the db. Then we're going to
provision a test environment, deploy the web and the
db containers to it. So now, we have a
system under test there. And we're going to
run a test against it. And so again, this
isn't an E2E test. this is just a
point-in-time test, but it does exercise both of
the layers of the application. It's just going to curl
the endpoint and grep for a particular string. This is a cookie shop,
so what it's grepping for is the words chocolate chip. That's a success,
you know, clearly. If it passes, then we're good. We don't need that
environment anymore, and we'll deprovision it. And if it doesn't pass, the test
is going to end and fail there. So most the time then,
the environment's going to stay around
so we can debug it, though there's some
trade-offs there that I'm going to show you. Now, in my experience,
the hard part of all this is the provisioning. There are a lot
of different ways to do it, a lot of trade-offs
among the different ways to do it. So we're going to look
at different mechanisms, different techniques
for provisioning and deprovisioning
a system under test. So the first one we're going
to look at is Docker Compose. Now, Docker Compose is basically
a wrapper around Docker. If you've ever run a Docker
Container with a dash d, you're creating a service
that sticks around. Docker Compose will spin
up multiple of those and let them network
in between each other. And in Cloud Build,
the way this happens is that when you run Docker
Compose within Cloud Build-- and there's a supported
builder for Docker Compose that lets you clone that and
run that into your project. And so the way that works is
that the builder instantiates these containers
that you're creating, this composed network,
as sister containers. They share the same network
as the build steps of your build so you could
interconnect between them. And it's exposed on a Docker
network called the Cloud Build network, and then other
steps can talk to them. So here's the flow. We start from our
checkout from Git. The gray box here represents
the boundary of Cloud Build. That's that security
boundary that Christopher was talking about where nothing
can get in or out without you expressly doing it. And it's actually out only. And so the steps here
are, we're going to build, run some unit tests. And then we use Docker
Compose to create those sister containers. And then we have our
integration test, which is just a shell script,
asking whether it succeeded or not. And our build is over. So the pros of this is
that it's fully ephemeral. Everything took place inside
that Cloud Build environment. And when the test is over,
everything gets torn down. You don't pay for anything
after the test is over, so there's no chance that you're
going to orphan those resources and be paying for them
after a long time. And for that reason,
it's cost-effective. You're only paying for
that Cloud Build run, which is billed by the minute. And there's also a
generous free tier, so you can get a lot
of mileage out of this. Now the cons are,
it's always ephemeral. You can't get back in
afterwards and go poke around inside the Docker
Compose environment. I've done things where I write
a bunch of logs somewhere, and it kind of works. But you're still a
lot of spelunking to go and figure out, well,
what happened in there? So that's not necessarily ideal. Limited fidelity--
because we're not going to use Docker Compose as
our actual production runtime. It's generally not going to
be your best choice for that. And so therefore, it's not
exactly the same behaviors. You don't know if
what works here is really going to work on prod. There are some funny
things about startup order. With Docker Compose, when
you start up the containers, they all just start. And if they fail to
start, they all just fail. And so when I was
working on this, I had the web app would
start up faster than the DB. The web app couldn't find
the DB, and it failed. I had to do some kind
of funky workaround to make it so that
it would hang around until the database came up. And in fact, Docker Compose
used to have support for health checks and
removed those deliberately, because it's not designed
to be a production runtime. And finally, the one that
I like the least here is the multiple configs. You can't use the same
config for a Docker Compose environment as for your
production Kubernetes environment. And inevitably, you're going
to have drift between those. So let's try again-- method 2. So in this case, we're going
to use a GKE staging cluster and deploy all of our
tests to that same cluster. So that sounds like a lot of
things stomping on each other, but we're going to
isolate via namespaces. And if you're not familiar,
a namespace in Kubernetes, it's an isolation
mechanism where you scope a number of resources
to a defined namespace. Now, namespaces can
talk to each other, but you have to do
that deliberately. So by default,
everything in a namespace thinks that it's
in its own bubble. And unless you work
hard to undo this, they're going to be
isolated together. In Cloud Build,
the way we're going to do that is, on
each build, we're going to create a uniquely named
namespace, deploy those things, test it. And then on success, we'll
destroy that namespace. Nice thing about namespace,
if you destroy the namespace, everything in it wipes
away along with it. So in this case,
what you'll see here is that we do our build
on our unit tests. And then we're going to
use kubectl to deploy a namespace into our
standing staging cluster. And you see, that's outside
of the Cloud Build network and, therefore, has a different
level of ephemerality. Then we run our test. If we get to the
end, kubectl's going to destroy that namespace
and all the resources in it. If we don't get to the end, that
last step isn't going to run, which means that namespace
is going to hang around so we can use any tools that we
can operate against the cluster on to go look into
those containers, look at their disks, et cetera. So the pros of this--
excellent fidelity, especially if your prod
happens to be running on GKE. And I hope you've heard
some things this week that have helped you think that
maybe that's a good idea. Easy tear-down--
delete the namespace, and everything's gone. And we have that ephemerality
that we're looking for that, on success, it disappears. On failure, it hangs
around to go debug. Cons-- imperfect installation. Now, it's unlikely, but it's
possible that one of your tests has runaway disk usage
or runaway memory. Messes up another test that's
running at the same time on the same cluster-- unlikely. But something that is more
likely to be a challenge is that you can't test
a cluster-wide resource. So for example,
the Agones project, which has presentations
this week, is the cool open
source framework for running multi-player
games on Kubernetes. The way it works is it creates
custom resource definitions. And custom resource
definitions are cluster-wide. So they can't test
different versions of a CRD all in one cluster. Also, if you wanted
to validate something against different versions of
Kubernetes, for example-- well, you can't have one cluster
that has multiple versions. The other problem is that
you may have orphaned forces that, if you fail a test and
you don't bother going in to look at that namespace
that didn't get deleted, it's just going to hang around. Not a huge impact. Having a namespace with no
activity isn't a major problem. But you probably would want
to have some sort of a cleanup script that runs once
a week or something and just deletes everything
that everyone forgot about. So now, if we wanted to
get absolute isolation, we can try using a Kubernetes
cluster for each test. And what we're
going to do here is we're going create a
Kubernetes cluster in a VM so that each test has
its own completely isolated environment, its own
VM that it's operating in. And to do that, we're
going to provision that VM with
Terraform, and we're going to install a single-node
Kubernetes application. There are several of these-- MicroK8s, Minikube. There's one called K3S. These are all compliant
Kubernetes servers that run on a single node. And in order to offset some of
the ephemerality issues here, we're going to use a
self-destructing VM. So a self-destructing VM is one
that, if you forget about it, it won't just hang around
and cost you money forever. Instead, it's built in
with a timer mechanism. And after a certain amount of
time, if no one touches it, it will delete itself. If you're interested in making
your own self-destructing VMs-- it's kind of fun-- check out that bit.ly link. So with this one, it's
similar to that GKE one, except this VM is being
provisioned on-the-fly and has Kubernetes or a
Kubernetes-compliant server in it. We're going to deploy our
application to that Kubernetes, test it. And at the end, we're going to
use Terraform to tear it down. And again, if it doesn't
tear down automatically because of a
success, it instead-- you can go and play
with it for a while. And it has an expiration date. It could be two hours, two days,
whatever-- it'll delete itself. So what's good about this one? Well, perfect isolation,
near-perfect fidelity. Again, these
non-Kubernetes Kubernetes, they are actually
certified compliant as being Kubernetes runtimes. But who knows? Maybe they're not
exactly the same thing. You can test those
cluster-level resources. And we've got that
ephemerality that we wanted. Downside, it's still
not exactly GKE. And also, there's
the possibility that these VMs hang around
longer than you want, even if they have an expiration. All right, finally, when I was
working on this stuff, people like Christopher
kept asking me, well, why don't you just create a
GKE cluster for every one? And I said, boy, that sounds
really expensive and slow. But of course, it's possible. And by doing that, you of
course get all the fidelity, all the isolation. You can create the entire
production environment. You could even,
if you wanted to, clone in a database or
other things from production so that every single
test has its own GKE. And it can be done. There's APIs for it. And I'm actually going to
show you as part of the test how that works. So let's check on that demo. Let's see. Here we go. So what I did when I kicked off
that build back in my IDE is, I ran what I call a meta CI. This is a Cloud Build invocation
that kicks off other Cloud Build invocations. And each one of those is one
of these testing mechanisms. In every case, we're testing
the exact same application, running the exact same
test script on it. But we're instantiating
and tearing down our test environment in different ways. So we have all four of the
methods I just showed you. And if we look
into the UI, we'll see that they all completed. And they all returned
the same result, but very different amounts of time. The Docker Compose
was very quick, because it didn't have to reach
to any external resources. But GKE cluster per test? Pretty slow, but
perfect fidelity. And the other ones are
somewhere in between. So what's the right
choice for something that you're going to do? It really depends on
what you're doing. Can we switch back to
the slides, please? So in summary, we took
a look at a number of different mechanisms
for spinning up and testing against a microservice
production environment. And you can see they
all have pros and cons. For example, with Docker
Compose, it scales really big. You don't have any
extra resources. That shared staging Kubernetes
is pretty good all around, but you don't necessarily
have perfect isolation. A Kubernetes per
test should scale. But at some point,
you're probably going to hit some quota
limits on IPs and things. I haven't tried running tens
of thousands of those, though. It probably would
be kind of fun. All in all, when I was
working on this, folks were asking me, well,
what's the right one? And I said, oh, it all depends. It doesn't matter
what you're doing. But really on balance
for most use cases, I think this is a really good
choice, this shared Kubernetes environment. If you turn on
auto-scaling, it's going to scale up and
scale down to meet the needs of your developers. They're not going to
have standing resources beyond a minimum number
of nodes over the weekend. You've got all of
that great fidelity. It's fast. And the isolation is probably
good enough for most workloads. If it's not quite
there, maybe you add one of those more intense
things as a final check. We can do more. We can use these
same kind of methods to do deeper end-to-end
testing-- a scripted test that creates an order, puts
something in a shopping cart, buys some cookies. We could do mobile web testing. There's a service, called
Firebase Test Labs, that Google provides
to test against Android devices and iOS. And we could use
external resources. So say for example we wanted
to spin up a dedicated Cloud SQL per test, we could do that. So this application and
all of these test methods are available on GitHub. I encourage you to
download it, give it a try. Pull requests are welcome. Feedback is welcome. Speaking of feedback-- wait,
I'll get to that in a second. Speaking of feedback,
tweet at me. Here's my Twitter. I'd love to hear from you,
what you're up to, how you're approaching problems like this. Christopher doesn't
really do Twitter. So if you want to get a
personalized response from him, just go to Cloud Build, and
go to that feedback link, and he'll write back to you. It's like a stationary
calligraphy. It's great. I kid. But it is actually a
really responsive team. And they absolutely want
to hear your feedback. And they're
absolutely listening. [MUSIC PLAYING]
