[MUSIC PLAYING] ANDREW FITZ GIBBON:
Hello, everyone. Now, I'm sure we've
all seen this happen. Your app crashes. Not only have we
seen it happen, we've watched it unfold in
real time and then really hoped the user sends
us a crash report. And these crashes take a
variety of forms, right? Maybe it's an application
not responding, a blue screen, a kernel panic,
or any of the many other forms of a crash. And we've all seen these happen. And the more devices
we end up running on, the more we'll run into
different kinds of crashes. App crashes happen. And, as much as
we, as developers, try to mitigate them, they
will continue to happen. So how do we actually see
them happen if our apps are deployed and out in the wild? How do we measure and observe
these highly distributed processes? Hi. I'm Fitz, Developer
Advocate with Flutter. And, early on in my
computing career, I was a sysadmin
responsible for keeping the servers and the services
on them up and running, 24/7. And, in my first
software engineering job, I was heavily involved in
the team's DevOps work. It wasn't called that back
then, but it was the same deal. Make sure our services
stayed up and deployments happened successfully. The common thing
between both of those? App crashes happened. It's definitely a thing. And, in both, I got
really comfortable looking at graphs
like this fake one, here, showing us how
well things are behaving. This graph might
be fake, but it's got the right idea, measuring
the health of our app. Here, the y-axis is
the percent of apps that don't report a crash. X-axis is the time,
roughly segmented by day. Everything looks
great here, right, a smooth straight line at 100%. Except, what is that on day 4? Alarm bells start going
off in my head here. That tiny little bump is
obviously a huge problem, but is it a crash? So before we get into the
code, let's clarify that word, "crash." If you're just using an app,
going about your business, then the app is not crashing. Right? Everything's working
and normal for whatever that means for the app, right? So, in its simplest
form, a crash is just anything happening
that's not normal. So, on one end of
things, we have normal. Everything's fine. And to understand if
something is a crash, that is, not normal, we'll have
to understand what normal is, too. On the opposite end,
we have those things that are obviously and
definitely a crash. Uncaught exceptions,
segfaults, whatever it is, the app is definitely
not working anymore. And, in the middle,
we have things that are slightly
different than normal. Maybe bad things
happen inconsistently or they happen
slower than normal. These kinds of crashes
are sometimes small enough for people to not notice them. My question, then, how do
we see things happening in the first place so
that we can figure out what to do later? To explore this, I
built a simple demo app. It starts with a Google
Sign-In and then flips to a randomly generated
grid of clickable squares. The blue ones are pretty boring. Click on them, and we get a
new page with a nice image. Go back. Click another blue square. Same thing. It's pretty reliable, and we can
go about our business as normal without crashes and keep doing
our perfectly normal thing of clicking blue squares. The yellow ones are similar
but slightly different. Sometimes we click
them, and the same thing happens as with
the blue squares. We go to a new
page with a photo, and sometimes it takes a while. And sometimes it
simply just crashes. Whoops. You might be able to guess,
by now, what the red ones do. Click a red square. App crashes. Restart. Click a red square,
and it crashes again. That is our definite
crash variety. What I'll do for the
rest of this talk is show how to monitor
these things, in the hopes that we can do something
about them later. And the reason that
this is a problem is that, yeah, we've got
our app running here. That's all well and good
when you've got one instance and it's on your dev machine and
you can attach the dev tools. In reality, your app
is wildly popular, and there are many, many,
many, many, many thousands of instances of it, and
you can't attach a debugger to any of them. As a former sysadmin working in
the scientific computing space, to me, this looks a whole heck
of a lot like a distributed system. And how do we monitor
highly distributed systems? With the cloud. And so I've got this app
wired up with both Google Cloud and Firebase, maybe
in a slightly unusual way, but today I'm going to focus
mainly on the Google Cloud integration. This app uses it in three ways. First, to get a sense
of what normal is. How long do various user
actions actually take to happen? In the demo app,
this would be things like the time between a
user clicking on a square to the image showing up. And a big part of that
is also just general logging and getting some
extra information about what the various instances
of the app are doing so that we can see
what's happening even without user interaction. And finally, of course,
is the interesting bit-- the outright crashes with,
if possible, stack traces. For this app, we first
sign in with Google to get that
authenticated client. And, on platforms where Google
Sign-In isn't available yet, you should use a lower-level
OAuth 2.0 client. But, in both cases,
we need to end up with an authenticated
HTTP client. And then, once at the point
of writing a log message, you should import the
relevant Google APIs package, in this case,
logging and trace, to access those
respective services. These are pretty
raw API clients. And so, for this case, I have
a small wrapper to help out. This wrapper does three things. First, initializes the resource
that we're working with. This defines, most
importantly, the cloud project to send the logs and
traces to and, second, it creates the log entry itself. The message log levels-- such
as info, critical, error, et cetera-- go here. And, finally, we actually
make the async call to actually write the log
message to the cloud service. Next, in order to log
uncaught exceptions, I've wrapped the
top-level runApp to both catch Flutter errors
as well as all other errors. These both call out to the same
logging helper we just wrote. Let's go turn this
into some actual code. Now that we have the client and
the basic logging mechanisms, let's go and actually use it. So here is where the various
square clicks are handled, a single onTap for
all three colors. The destination is the same,
so let's use the same function. Obviously, things are a
little bit contrived here, randomly forcing different
unique exceptions to be thrown in
order to guarantee that they're not caught. But my first step, here, is
to get a sense of normal. How long does it take between
tapping a button and the dash image showing up? So, in my helper, I'm
going to add some tracing. If we don't already have
a trace and a span ID-- these represent the
operation that we're doing-- we should first create one
for this instance of the app. If we already have one,
we can just reuse it. And then we can add
it to the log entry to get some of those timings. I also want to
differentiate between what's part of a trace
versus what's just a general log, that second thing
that we wanted to do with cloud logging, and so I'll add
another Boolean parameter here to say that. I'll then have to go back
to the calls of this helper and add this parameter. When the button is clicked,
for example, no matter what the color is,
log it as a trace. Second, when the new widget
loads with that image, we'll send another trace log up. That'll give us the difference
between the two timings for how long that
operation takes. And, for all the
other logs, these are just the general logs. Right now, they're
going to be not traces. Once we have that
up and running, we can open up the
logging console and see how it shows up. I've pre-generated
a bunch of data here so that we can
see a sense of normal. And, here, it looks a lot like
that fake graph of mine, maybe just reverse. It's a histogram of the various
logs colored by their level. Most are just info, so we see
that steady stream of normal, those light blue bars. And that trickle of
red, here at the bottom, represents the yellow
and red button presses, those critical and
error logs respectively. Going back and
looking at the traces, we can also get a sense for the
timing of these user actions. Most of these take a
reasonable amount of time, but some are quite a
bit slower than others. Curious. These metrics give us a
good idea of what's normal. That's our baseline,
the most common duration between these two widgets. Now, this may be an instance
running in Debug mode, but we can still click a
few buttons here and there. Click some red ones,
clicks some yellow ones, click some blue ones, and
go back to the console and see, for the
outright errors, if we click through
to a specific entry, we should be able to
see the stack traces that I added earlier. And there's our logs, and there
are some minute traces, too. Now, as mentioned
earlier, in reality, you're likely to have lots of
instances of this app running in production. And, under normal circumstances,
everything works great. That's why we call it normal. And so we don't
want all of those to be constantly logging
near useless metrics. And so, with Firebase,
I've pulled it into this app to handle some of
that fire hose of information. Using Remote Config,
I've set a value to control how much gets logged. And once we have that
baseline understanding of what normal
looks like, we can use that Remote Config
value to pull back on the logging frequency. In the code, there are
three things to note. The Firebase client is
using those same credentials that we created for
the logging service. For the Config
client, we also have to set some default values. In this case, log everything by
default, unless told otherwise. And then, within
the logging helper, I check the Config client to
see if there are actual values there for us to use. Since this is just
an info-level log, I might not want
to always log it. The Config value will
tell me if I need to and, if I shouldn't, just
skip everything else. That way, I have now
limited the rate of when this app sends logs, if needed. OK, throughout this talk,
I've shown a few things but, most importantly,
defining a crash as anything that's not normal. And to understand
what not normal is, we first needed to
understand what normal is. To do that I, used Google
Cloud's logging service, treating our app as just
another distributed system to get a feel, with numbers,
of what that actually is. After that, I looked at how
to use Firebase to help limit some of that fire hose of data. Under the vast majority of
cases, everything is fine. Once we know the
baseline of normal, we might want to dial down
that logging frequency so that we don't exhaust
our logs quota too quickly. That's been me watching
a Flutter app fail, but with logging and
metrics behind it. Thanks for watching.
