[MUSIC PLAYING] DANNA KELMER: Hello everyone. Good morning. My name is Danna. I'm a software
engineer on the Bazel Team at Google in New York. Today, we are going
to talk about problems that certain organizations
start to encounter as their company and
their projects grow, in particular with-- in
relation to build and test times, and then go over
the ways in which the build tool Bazel can help
alleviate some of those pains. And finish up with a testimonial
from the Angular team about how it's been
working for them. So why might an
organization need Bazel? Certain problems start to arise
as your organization grows. Your projects start
to grow and the needs of your infrastructure start
to suffer with the load that it requires, and in
turn, this might affect your-- the development
experience of the people at your organization. And in part, this
happens because, as you have more developers, they
start to make more changes. And more changes means that you
need to run all of your tests far more often. So if you wanted to run all of
your tests before every submit, that starts to increase
as you have more people. The second thing that affects
this is as the feature space starts to grow, this obviously
increases just the sheer number of tests that need to run. So this is more load that
must happen with every change. And the final part is as
your project becomes larger, each individual integration test
requires a ton of compute power just for a single one. And so it starts to
become, very quickly, infeasible to run all of
your tests on every submit. So basically, many
organizations tend to give up on integration testing. And this means that
instead, they'll break up their projects into a
number of microservices where each service is responsible
for its own set of tests, and then those will
run on every submit. And you'll only ever run your
substantial integration tests when you attempt to release. So this is an example
of a way that you might break up some project. And what this means is
that when you're releasing, that's the first
time that you're going to encounter problems
with the interaction between these different
microservices that have been working separately. All right, so what is Bazel? And how is it going
to help this problem and enable and empower
your organization to run all of these
tests as often as possible on every commit? So Bazel is a free
and open source automated build and test tool. This tool has been used at
Google for nearly the past 15 years to build all
of its source code. And I'm going to go over
the properties of Bazel that enable this, that enable
you to run all of your tests across your entire organization
as often as you'd like. So the first property is
that Bazel is hermetic. This means that Bazel
is aware of all inputs before the build begins. So of course, this includes
all of your source files and everything that
would need to be built. But this also includes
any tool, for example, that would need to be used
over the course of the build. And in turn, this
hermeticity directly lends itself to Bazel
being incremental. And so because Bazel
is aware of all inputs, for any change you make,
Bazel can very easily calculate the minimal
amount of work that needs to be done in order
to create your finished build product. And so, if you make
a small change, you should expect that
the amount of work that Bazel will do
will be proportional, that it will be small. So I'm going to
demonstrate that. All right. So here's an example. We're starting. We're asking Bazel to
build all the tests. And, just to start, we see that
everything is fully cached. So this is imagining a scenario
that you just did a build, right? We're starting. Everything was already
built, and now we're going to make just
a small change. So the change is going to
be to a single test file. So a test-- a test itself, often
nothing depends on that test. So the expectation is that
when you make the change to this single test, that only
that single test will have to be rebuilt and retested. So here's an example test file. We're going to add an additional
assertion to this test to make sure that everything,
or that this thing is not null. And then, again, what
we're going to do is we're going to ask
Bazel to retest everything. But the expectation is
that Bazel will be smarter, and it will only
rebuild that single test and rerun that single test. So Bazel is working
and beautiful. Everything is cached. But we see it executed
one out of 174 tests. So that's the one
test that we edited. Everything else is still cached. All right. The next property of Bazel
is that it's deterministic. So Bazel uses the checksum
of all of the inputs to the build in order to
determine whether or not they've changed. And what this means then is that
Bazel can very safely cache any intermediary build
artifacts that it produces. And then instead
of redoing the work to do it, if it knows that none
of the inputs have changed, you can safely retrieve
the output artifact from this cache. And this is a huge
opportunity for optimization at your organization
because you're not bound to keep this cache local. You can share this
crop-- this cache across your entire organization,
and all of your developers can populate this cache, and
not only your developers, but your CI system can
also populate this cache. And then whenever your
developers are doing builds, Bazel can then check
the cache and see if the artifacts that it
needs are already present. And if yes, simply retrieve them
without doing any work at all. So this is some example of how
this cache might be shared. And of course what this
means is that, for example, the first built of the
day will also be fast. If you say have your
cache, and it's-- sorry. If you have your CI system
populating this cache, then when your developers
get to work in the morning, they do their first build. That build will probably be
trivially fast because almost all artifacts will be cached,
very similar to the example above before I change the test. All right. And this cache can be anywhere. So you can use Google Hub-- Google Cloud Storage in order to
have this cache be in the cloud not hosted by you. But you can also
set up this cache to be hosted on premises
for everyone to use. And the final property of Bazel
that we're going to go over is that it's fully cloud ready. And so because of all
the previous properties that we talked about,
its hermeticity on its determinism,
not only can you store each of the
intermediary artifacts, but you can also do
the work on the cloud. All of the build actions
can happen in the cloud. So we're going to look at
an example of another build. So in this example,
in this example, as before, we're imagining we're
starting from a clean slate. Everything has been cached. But this time, instead
of changing a test file, this is editing a
source file that is at the very bottom
of the project. Nearly everything
in this project depends on this source file. And so, now you're going to
expect a full cache miss. You're making a unique
change to your project. And so there's no way that any
artifacts will be in the cache. And so Bazel will do
its incremental build, but the incremental build
still is a lot of work. And so it's going
to start working, but now you're bound by
the local compute power that you have. So we're actually
going to let this keep running in the meantime. But as we said, these builds
can be moved to the cloud so that you're no
longer going to be bound by the compute
resources that you have. So, to demonstrate that,
we have this project. And similarly as
before, we're going to start from a clean
fully-cached slate, and then we're going to make
the same edit to the same files that we did before. And again, this is a file
that everything depends on, so Bazel will have
to do all of the work to rebuild all of the project
and rerun all of the tests. But then when we invoke
Bazel in a moment, we're going to see
that we're going to pass an additional flag
that tells Bazel to execute all of the build actions remotely. So this is a setup that has
been done on this project. There's a bit of configuration. And then you pass
this additional-- that doesn't work. Another dash dash config
equals remote flag, and then Bazel knows to
run these actions remotely. And if we see here, we
can see that 150 actions are running simultaneously. So this project was configured
with 150 remote workers, and Bazel is
utilizing all of it. And this means it's
running as many tests as the configuration allows for. So now what happens
is now that we see that the number of
actions running simultaneously is dropping, that's simply
because there aren't any more tests left to run. And so at this
point, you're simply bounded by your
slowest running test. So let's just go
back at this point, 142 tests have been
executed so far. We go back to the job
that was running locally. We see that only 20 tests
have been executed in the time that we've been working. A couple of these tests
take a while to run, so we will move on. But we see 154 have
finished so far, and they're all
running simultaneously. So 14 are still running. All right, so that
was a demonstration of how Bazel will
run in the cloud. And now I'm going to pass it on
to Alex to talk about Angular and how they've been using it. ALEX EAGLE: Thanks, Danna. [APPLAUSE] Hi, everybody. My name is Alex Eagle. I work on the Angular
team at Google. I'm the lead for our tooling
team, which includes things like making the tools that our
Angular users, about 1 and 1/2 million of them, use to
develop their applications, as well as supporting
our own team and our internal
development velocity. And I want to tell you
about how much Bazel has improved our lives. If you haven't heard of Angular. It's a enterprise-scale
web framework for a building very
modern web applications. And it is a big project. So how big is it? So as Danna pointed out,
the development velocity on your project, it
depends on how many people and how big the tests are. And you might think
that you might-- we need to break up your
projects so you don't run all the tests all the time. That's really not
what we wanted to do. So we wanted to use
Bazel so that we could run all of our
tests all the time and keep our development
velocity high. And our numbers, I just
grabbed these off of GitHub, and in the last
30 days, we've had 92 authors, over 400 commits
touching over 1,000 files. So this is, I would say,
a pretty medium scale. I know a lot of enterprises
have much bigger apps than this, but it is enough of
a scale that we saw the problem the Bazel solves. So this is what our continuous
integration time looks like. You can imagine a developer
on the Angular team, one of those 95 people, having
to wait an hour for all of the tests to
run, and then they find out something was broken. And so you can imagine going
through this round trip many times. This significantly hurt
our development velocity. And I want to point out
here that the numbers I'm showing here are non
incremental, so this is just like the uncached build. That's sort of the
detail of the fact that we get contributions from
the open source community, and so using a [INAUDIBLE] cache
has some security risks for us. And so this graph is
showing that 60 minutes is before we enabled the
remote build execution, the dash dash config equals
remote that Danna showed. And today, I just
checked this morning, and a typical round trip
time is 7 and 1/2 minutes to run all of the tests. So this is a decrease
of 87% on our test time, and it has made a really big
difference to the Angular team. There are other
features of Bazel that have helped us out a lot. This is easily
worth the investment that we put into migrating
our build to Bazel. So this talk is about
developing like Google, and the Angular
project itself is just a small part of all
of the Angular work that goes on at Google. We have a lot of customers
internally, over 600 of them, and some of these applications
you've probably used. And all of these applications
rely heavily on remote build execution to allow a
very large number of Google Developers to be productive
working in these applications. And yes, we write full
stack integration tests and expect them to
run all the time. So now that Angular itself
has been successfully ported to Bazel, we want to make
it possible for all of you, if you're using Angular
2.0, to the same thing. And so as-- we're developing
an opt-in preview which will ship in the next month as
part of version 8 of Angular, that will allow all of
our enterprise users to be able to opt-in
to using Bazel. We will plan to work with
them over the next six months to help them find patterns of
porting parts of their builds to run under Bazel
and allow them to use remote build execution. And I also want to point out
that our plan for version 8 is that, if you don't have
a very large scale project, there's actually no
porting required. Bazel can just drop
into your project. It's really for the
enterprise use cases that some extra migration
work will be needed. So there's one more thing
I want to talk about. We showed how build
and test can benefit a lot from incrementality
and can be run by Bazel, but we're at the
cloud conference, and we are all thinking
about deploying services. So let's talk about
deployment for a minute. A year and a half ago, when the
Angular team switched to Bazel, we also switched how we
do our own deployments. And so, we run basically a loop
over our packages of just doing Bazel deploy of each one. That's not that interesting
because we're a library. And so our deployment
is a shipping an artifact to a registry. What's more interesting
is that we have an example application that looks like an
enterprise Angular app would, and as Danna showed,
only the change packages get rebuilt when we do a deploy. So Bazel's smart enough to
know what work it needs to do. But then, the changes
that are produced by that can go into a new layer
in the docker image. And then only that
new layer needs to be uploaded to the Container
Registry because, of course, Docker has the property that
each layer has a checksum, and if the checksums match, it
doesn't need to do any work. So the incrementality benefits
can go all the way from your build through to
your deployment. And so I have a quick
video showing that. So this is our
example application. And you see there's
some text there. The live edit and deploy. And so we're going to make
a change to that text. I should point out
while this is running that all the demos
you've seen today were actually Angular itself. And so those 174 tests
Danna was running was the Angular repository. That was a demo of our
actual day-to-day development experience. So now we're going to
run deploy dot replace. So the deploy target
is a Kubernetes target. This is going to go
to Kubernetes Engine, and the replace is just a verb
saying what we want to do. If we were making
a new application we would say create. And so Bazel, as you
can see, 13,000 actions didn't need to be run. So we got a lot of
cache benefits now. And then the remaining
part, the work that's still happening
here, is we're bundling up the application. We're running some slow steps to
do the production optimizations for this application, and
it is a medium scale app. So that takes some time. And then once the
app is fully built, Bazel knows how to construct
the Docker image for it. And so now we see this
image layer is running. It's running too quickly for
me to talk over, which is good. And then the gzip happened,
and so now there's a kubectl command running. This is the command line utility
that deploys to Kubernetes. So again, I did all of
this with one command, and I didn't need to say what
needed to be built first. I just said, just
like redeploy me. I don't know what changed. So now we're going to
wait for Kubernetes Engine to spin up some new
workers with the new code. There's a load
balancer in front, and so now that LiveEdit
shows up in the application. So that's all we have. We're happy to answer some
questions after the talk. We'll be over here in this area. Thank you very much. [APPLAUSE] [MUSIC PLAYING]
