[MUSIC PLAYING] NICHOLAS CLINTON: All right. So welcome to Deep Learning with
TensorFlow and Earth Engine. This is kind of a follow-on
from the previous talk that was downstairs in Endeavor
with Noel and Emily. I'm Nick Clinton. And we also have-- ALEXANDRINA GV: Hi, I'm AGV. We're both developer relations
engineers for Earth Engine. Just a quick shoutout-- David Cavazos is another
amazing peer of ours who wrote an end-to-end
sample that you will be able to visit today. But he's not on stage, so just
a quick shoutout to him as well. NICHOLAS CLINTON: All right. Thanks. One thing I forgot-- get this link. tf2_g4g_22-- that is the
link to these slides. There are tons of other
links to Colab notebooks and Docs and code editor links
and so forth from the slide deck, so you're probably
going to want that link. tf2_g4g_22-- somebody
remember that. ALEXANDRINA GV:
Or take a picture. I'll also show it at the end. NICHOLAS CLINTON: All right. Let's just dive right in because
we've got a lot of content here, and we're going
to try to get through it all without going over. So we're going to talk about
a bunch of things here. The first is just
options for getting training data out
of Earth Engine and training a model with it. We'll talk a little bit
about what deep learning is and the architecture
of models that you would use to implement deep
learning in TensorFlow. And then we'll have a short
demo of some working land cover classification examples. And then we'll just
wrap up from there. OK. So as I mentioned,
this is a follow-on from the previous one. If you weren't in
Endeavor this morning, those slides will be online. But that's sort of a gentle
introduction to machine learning with Earth Engine. There are lots of options
for training a model. Oops. OK. There we go. Options for training--
lots of options for training a machine
learning model in Earth Engine. And we like to use a
lot of food analogies. So the built-in Earth
Engine classifiers, like the RandomForest models
that they were talking about earlier, are not very spicy. Noel showed you
eight lines of code. You can implement a RandomForest
in eight lines of code. But if you've got an
appetite for spicy food, you might love TensorFlow. And that's primarily what we're
going to be talking about here. You can build much spicier
models with TensorFlow. So again, basic stuff
built into Earth Engine with any machine learning or
data mining problem-- often you start with a
simple model that lets you understand the data. That's a great use for these
Earth Engine classifiers. These include RandomForest,
CART, NaiveBayes, Support Vector Machines,
and unsupervised classifiers or clusters. These are definitely very easy
to use and get running quickly. But there are
options for training and predicting in the Cloud
if your model gets too large or your training data
set gets-- whoops, sorry. Here, I got it. OK. So another food analogy-- let's say you want
to have a cake. You can just go to the market
and buy a cake off the shelf. That cake might be fine for you. But if you're good
at cooking and you want to make a really fancy cake
or maybe a gingerbread house or something like
that, you've really got to build it yourself. And so it's your
appetite for building these complex models will
determine which route you want to go. So when do you want to
build the custom model? In other words, when
do you want to graduate from the built-in Earth
Engine classifiers and use something fancy? If you are using any of
the softwares on the left-- so if you like coding
in Keras or TensorFlow or some other popular machine
learning packages that are supported by AI
platform or Vertex, we have connectors for those. So that's an excellent reason
to want to build these models. If you want to build
something very fancy-- like, for example, Dynamic
World was just announced. Chris and Val in here, they
built an incredible model. And it's very fancy. So if you want to do
something like that, that's another great reason
for wanting a custom model. If the size of
your training data set gets very large,
for the reasons that were mentioned
earlier this morning, Earth Engine can have a problem
with those very large data sets. They don't fit into cache. It can take the model
a very long time to train on those
things-- longer than the five-minute
code editor time-out. So this is another
excellent reason to want to go to a custom model. Did I just skip one? This thing is-- I got to use these buttons here. OK. What happens if you
have too much data? You can get an error. So as Mo mentioned
previously, those tables may or may not fit into memory. And it depends on
not only the number of points that you have in the
table or the number of rows, but also the number
of features-- feature in the machine
learning context, not to be confused
with EE dot feature. OK. If your table gets too
big, if you can't export it in one single task to get
it to some other location where you train
your model, if you start looking at hacky
solutions for exporting your data in a bunch
of different pieces-- that can be a problem
because you can only run so many tasks at once. So these are all
excellent reasons to think about going
to a TensorFlow model. OK. Just to summarize,
built-in classifiers-- fast and easy to use. Custom models--
bigger training data sets and more complex models
with longer training times. Another great reason to want
to build these custom models is if you want to implement
a convolutional model. So in the left here, we have
sort of a representation of a pixel-based classifier. So each pixel is considered
differently by the classifier. And then each answer is, too. But patterns are important. And these might be the
patterns of rivers, and roads, and landscape features that
are relevant to the classifier. You'd want your classifier
to see those things. And it's more efficient for
models to learn those patterns than it is to try to figure that
out from a single pixel basis. ALEXANDRINA GV: And that's it. NICHOLAS CLINTON:
The weather model? Should I talk about that? ALEXANDRINA GV: I can say it. NICHOLAS CLINTON: OK. AGV, take it away. ALEXANDRINA GV:
Yeah, so just as we move on from one
topic to the other, the analogy here
is if you actually are trying to do any kind
of weather prediction, traditionally you're
simulating every single atom. And that can get pretty
big computationally and very inefficient. But if you instead find
patterns, it's a lot better. And we have better models these
days for weather prediction. So following that
analogy, we'll dive into more about deep learning,
which is finding patterns. So out of all the many
ML algorithms out there, I'm happy to share that deep
learning or artificial neural networks is a technique
that can be used for almost any supervised learning job. In supervised learning,
you tell the computer the right answers you're
looking for through examples. And deep learning is flexible. And it's a very general
purpose algorithm, especially for
things like images, audio files, or
video files, which are multi-dimensional data. A training model that classifies
tree species, for example, using satellite images is kind
of like an image segmentation problem where every pixel
is classified in the image. However, there's more bends than
the typical three-channel RGB images. But deep learning approaches
things very differently. There's no writing a function
with explicit and sequential steps that reviews every single
pixel one by one for the image, as traditional software
development is done. So for example, in
traditional programming, if you wanted to add
logic to classify a new type of true
species to existing code, you can actually end up with-- let's see if the GIF
starts from the beginning. But basically, yes,
so you're trying to add a new type of tree
species to existing code. And then you start
having a bunch of bugs because there's a lot of
dependencies from your older code that you didn't really
change from your instructions. And so at the end
of the day, you end up very
frustrated and having to rewrite your entire
function from scratch again. But with deep learning, you
can really simplify things because you don't
have to handcraft every single instruction. Based on that same
example, you can actually give a computer images
with tree specie labels and let it learn
from those examples. And when you need to
add a new tree species, you just give it new images
and retrain your model for that species. So now that I covered
what deep learning-- what it can help
with and why it's more advantageous than
traditional programming methods, I'd like to move on how
to approach a problem using it. So let's say we would like
to measure deforestation using deep learning. Like many machine
learning algorithms, it only understands
tensors, which are numbers stored as an array
or vector in one dimension, or a matrix or table
in two dimensions, or as multi-dimensional
data, which I was talking about images, video, et cetera. So with that in mind, we
convert satellite images into tensors, where each
pixel has one number assigned for every satellite band. And to get started with building
a model, we need, obviously, a data set that includes
satellite images with an even amount of labels marking where
there are trees and where there aren't. The next thing is
we need to be really clear on what is our goal. And there are a
few common goals. Regression predicts a number. Binary classification
makes a prediction between two categories. Multi-class classification
helps you classify things across multiple categories. Semantic segmentation classifies
every pixel in an image. And in our case, we know we
want to know if there are trees or not in every single pixel. So we're going to do binary-- two-- semantic segmentation. So based on that goal,
we expect the outputs to be the percent
of trees that are in every pixel between
a number of 0 and 1. So you can imagine, 0
represents no trees, and 1 represents that I have
a high confidence in trees. But how do we go
from input images into probabilities of trees? So think about it this way. There are many ways to
approach this problem. You'll notice that a lot
in machine learning-- this kind of brute
force, you got to try all the
things, et cetera. But to keep it very simple
for you in this session, there are three
common ways to do it. My peers and I prefer a
fully convolutional network when building any kind
of map with predictions. Why? Because they take
neighboring pixels and they generate context for us. Remember those having to write
every single instruction? Forget about that. It takes a context
from the images automatically, kind
of like our eyes do. And since a model is a
collection of interconnected layers, we have to come up with
an arrangement of those layers to transform our data based
on our desired output. Each layer, by the way, has
something called an activation function, which performs
the transformations at every single layer before
it passes on to the next layer. And my peer David's
example, by the way, walks through
everything about this. We're trying to keep
it very high-level to make it accessible. But I'll share
the link in a bit. And so in regards to our
deforestation use case, a good starting point is
to have three layers, which includes a layer to normalize
your data, a 2D convolutional layer, and a
deconvolutional layer. Then you reach your
fourth and last layer. And depending,
again, on your goals, at the beginning
we choose something called a loss
function that helps us score how well the
model did during training. And FYI, again,
since a lot of this is kind of brute
force experimentation, we wanted to put a nice
little table for you to visit, depending
on your goal, which activation function to use
and which loss function to use if this is helpful for you. You can later check this out. Or if you're watching
the recording, you can pause it or
check the slides. And again, after
choosing your layers, you will split your data into
training and validation data sets, which I'll talk more
about in a little bit. But just remember
that all of this work is about just experimenting
repeatedly until you reach your desired results. All right. So overall, that was
a super quick summary to help demystify
neural networks. I want to now dive into the
products and the process of training a deep
learning model and hosting it to
make predictions. So to get started, you
will need an account with Google Earth Engine-- which, a lot of
you know, is free for non-commercial entities--
and a Google Cloud account, which has a free tier,
if you're just getting started, for any user. I wanted to break up the
products by function first. So we have Earth
Engine and Dataflow. And they're both data
processors in this case. Dataflow is a fully
managed service that executes Apache Beam
pipelines on the Google Cloud platform ecosystem. And then next, we
have TensorFlow Keras, which is an ML
library, and Vertex AI, which is a machine learning
platform on Google Cloud. And a quick note
on that is I want to point out that Vertex
AI is the latest ML platform for Google. Some of you may be familiar
with its predecessor, Cloud AI Platform. If you are currently using
AI Platform, no problem. But in the future, as
you're creating new models, the product team recommends
to use Vertex AI. AI Platform is currently
integrated with Earth Engine, and we are currently working on
the integration for Vertex AI. Then there's Cloud
Storage, which is a bucket that we use
to store and move data across the products. And just as a quick fun
fact about storing files, depending on the ML
framework that you use, you will typically
work with TFRecords if you use TensorFlow, and
you will use NumPy files for other ML frameworks. Then there's Cloud Run. And if you've heard
of App Engine before, they're kind of siblings. And they allow you to host any
service, including your model. And finally, there's
Earth Engine, again, to visualize your
predictions, or Colab, which is a notebook created by
Google research team, where you can use a map
library called Folium or any other type
of mapping library to be able to visualize. All right. And this, by the way, is just
a quick screenshot in case some of you have never seen
what Colab is, that notebook. You can write text in markdown. You can click Run
on snippets of code. You can enable ML
accelerators like GPUs. And it's very collaborative. You can share it as a link. All right. So as we put all of
these products together, let's move on to
the architecture. So we built this
design keeping in mind the inputs and the outputs. And our inputs are Sentinel-2
images and elevation. And our outputs are land cover. And sure enough, what
we're trying to do is land cover classification. If you want to take a
quick picture of this or later visit it
in the slide deck-- but basically, this is that
end-to-end sample that my peer, David, made just for y'all
for this event, especially, that takes you all
the way, all the code, all the way from starting and
understanding terms in machine learning to visualizing
it in Earth Engine, using AI platform. Is that OK? Yeah. All right. Yeah. After we establish what our
inputs and outputs and we now want to pull that data
from Earth Engine, and we use that with Dataflow. And we get a collection
of input and label pairs. And we save them as TFRecords. And the reason why
we use Dataflow, by the way-- we could
use just Earth Engine. But we notice that if your
data set is very large, it can take days, and
luckily maybe hours. So if you really want
to reduce it to minutes, Dataflow is your
best friend for that. Next, we save those as
TFRecords into Cloud Storage. And then we split our
data into two data sets. 90% is for training
the model, and 10% is for validating that
it learned from examples it has never seen before. And then we train
that in Vertex AI. Note, there's no
hard and fast rule if it should be 90/10 or 80/20. You get to choose that. We just chose 90/10. Also, note that the process
of splitting the training and validation data set
can be at different stages, depending on what machine
learning framework you use. So just in case
you're like, I've seen it at this stage or that
other stage, it's up to you. There are pros and cons,
and the notebook actually talks about that. If you also want to
not use Vertex AI and train on your computer,
you could do that. Just know that,
depending, again, on that size of your data, it
could take a very long time. All right. So after we train the model,
we store it into Cloud Storage as a TensorFlow file
called Saved Model. And we can then host
that model into Cloud Run and begin to classify
incoming land cover labels. So you'll see that there's a
difference between training and hosting the model. Now, in the event you want to
eventually change your model-- you want to make some tweaks-- you can use Dataflow
to do that in batch. And it'll change all
your historical labels. And then you can go back to
using Cloud Run for streaming those predictions again. You can also host the model
locally on your computer or Colab notebook, by the way. So there's a couple of
options for that hosting part. The most important question
for a lot of folks is price. So I just wanted to leave
you that we have a-- I co-created a YouTube series. Every episode is
about 10 minutes. We just dive into
machine learning, usually with Earth
Engine and Google Cloud. And of these recent episodes,
we've noticed as of October 22, 2022, it's cost us less
than $5.00 to do that from end-to-end. However, the reality
is that most of you want to connect a model
back into Earth Engine. And so I wanted to just
share two more steps. The first is, instead of
hosting it in Cloud Run, you can actually host it into
AI platform, which is integrated with Earth Engine. Just know, though, that
it's a 24-hour paid service. And so it can cost you
upwards of $100 or more a month to host your model
to stream predictions. AI platform supports TensorFlow,
Scikit Learn, XGBoost. And it can also do PyTorch if
you use a custom container. Also, just be mindful
of the zoom level because that can also affect how
many requests you're sending. Now, a cheaper alternative
to AI platform, although you will not
have the convenience that AI platform offers,
is to manually translate the model's output,
which is NumPy Arrays, into Cloud-optimized geoTIFFs
in order to load it back into Earth Engine
using Cloud Run. So there's a little bit
more of a breakdown. So within this web service,
you store the NumPy arrays in Cloud Storage. And then you spin up
a container by using GDAL, which is an open
source geospatial library, to convert the Cloud-optimized
geoTIFFs into Cloud Storage again. And this way you can view
predictions from Earth Engine or from your browser. And by the way, we
have a sample coming. We know that's very
important for the community, so stay tuned. We're working on it. And this is also
another resource that I'm not going to go through
right now because it's just a lot. But if you're trying to go
through branching logic of what to use based on your
use case, that's my attempt at providing that. All right. Now I'll pass it
on to Nick again, who is going to give you a demo. NICHOLAS CLINTON: All right. Thanks. How are we on time? Not too bad. How many people
are checking email? None. Sweet. Oh, one? Couple. All right. Whatever. That's OK. All right. So as AGV was saying, we've
got this end-to-end notebook that really enumerates every
possible combination of Cloud Services and data extraction
methods that you want to use. There's a link to the
previous version in here. Next version is coming soon. Those were both done
by David Cavazos. And he and I had
dueling banjos on this. So the examples I'm
going to show you were simplified
versions of that. OK. Step 1, get stuff
out of Earth Engine. Get your training data
out of Earth Engine. Option 1 is you export tables. And there are lots
of docs on that. See the TensorFlow
page for more info. There's the REST API. So if you want to make calls
directly to the REST API and just get blocks of pixels
as NumPy arrays or JPEGs or whatever you want
to do, you can do that. And there's an example
for that in our docs. You can get patches
with getDownloadURL, which is somewhat different
than what we've done previously. And you can scale that
with Dataflow, which makes it a really nice option. Coming soon, there will be a
client-based compute pixels, which is even lighter
weight than getDownloadURL-- not quite ready yet,
but coming soon. OK. So example number 1 is a
fully convolutional model for semantic segmentation. We want to go from
some set of input bands to a label in every pixel. And how do you do
that with TensorFlow? This is the demo
Colab that's going to run through this
example end-to-end. So you can grab that Colab. It's not going to work
right out of the box because you have to substitute
your own Cloud project and your own Cloud
Storage bucket. But once you do that,
all the code should work. And please, let me know
if that link doesn't work or if any of the
code doesn't work. All right. So this is the goal here. And the first step is
to sample some patches. So in the morning session,
if you were in there, you saw how to sample pixels. And Noel showed you
some workarounds for getting neighborhood
information in there. You can use neighborhoodToArray
or neighborhoodToBands or compute gray-level
co-occurrence matrices-- that kind of thing. But you're not going to
get patterns that way. And so the idea
here is that we want to sample patches out of here
that contain spatial patterns. And we're going
to do this as 128 by 128 patches with nine
bands or something like that. And these are just some
regions that I drew by hand. We're going to
sample 128 patches from each one of those images. Oh, by the way, where is it? I already revealed
it on the next slide, so I forgot to play that game. Sorry. All right. It's near Ho Chi
Minh City in Vietnam. And so we're going to
go into some code now. And I'll just walk
you through this. The idea here-- oh, do
I have a laser pointer? Sort of. OK. The idea here is
that you want to take a sample of random locations
from those polygons that you saw on
the previous slide. And an easy way to do
that is stratified sample. That will give you approximately
the same number of points in each class, which is good. The landscape distribution
may be very imbalanced. But stratified
sample will attempt to get you a balanced
training data set. And then at the end here, all
we're doing is calling gefInfo. First, we convert
the sample to a list, and then call
geInfo on the list. And what that does is
that moves the sample from the server to your
client, which, in this case, is the Colab notebook
virtual machine. Once you have that
sample on the client, you can iterate
through it or make it into an iterator or a
generator and yield each set of coordinates one at a time. So this thing is just a function
that yields coordinates one at a time from those boxes. Now, we get the set of points. And what we want
to do is we want to get a patch under the points. So one way-- in our docs, we
show you neighborhood to array. And then you export that. Here's another way. This way uses getDownloadURL
on your image of predictors. So the image that
you saw on the left, which had spectral
data from Landsat-- I think it was Landsat. No, maybe it was Sentinel. I don't even remember. Anyway, you get
spectral imagery. You call getDownloadURL. At your point, you tell the
size of the patch you want. You tell it you
want a NumPy array. And that's it. You can use whatever
request library you want. It's a GET request to
the Earth Engine service. And then you load the result
into a structured NumPy array. And all this is
happening on your client. OK. So the advantage
of this is that you can connect to the high volume
API and send a lot of requests. And if you're using tasks,
those tasks can get too big. As AGV mentioned,
they can take days to finish, which is
not always ideal. And you can only run so
many of them at a time. So this gives you a
potentially more scalable way to generate a lot
of training data. Next, please. Yeah. Thank you. Oh, can you go back one? Sorry. The other thing
that I'll mention is that this function
is retry-able. Batteries are dead. The function is retry-able-- the top line there. What that means is that you
only get a certain number of requests at a time. When you run out, you
have to wait for one of the previous ones to finish. And then you can do another one. And by decorating this
function with Retry, that will happen for
you automatically. So that's another advantage. Next. OK. This is what it looks like. This is our patch
of Sentinel-2 data. And it has a shape of
128 by 128 by 9 bands. These are the bands that we're
going to use for prediction. There's nothing
scientific about this. I'm just using these
bands for demo. And then-- whoop. ALEXANDRINA GV: I'm so sorry. NICHOLAS CLINTON: Previous one. OK. Thank you. The labels are from WorldCover. And so all we're doing here
is we're just getting labels from an existing
land cover product and using those
to train a model. If you're doing something
fancy, you probably want to collect your
own labels somehow. But we assume that you
have a set of inputs and a set of labels. And you want to predict
those labels from the inputs. Next, please. OK. Before you can send those
patches into TensorFlow, you have to serialize
them, or at least this is an efficient way to get
those things into TensorFlow. And so the point that I'm
making here is that you-- so the array that you
get from getDownloadURL is a structured
NumPy array, which means that you get the bands
out by just using their name to index into that array. So you get the name
from the array. You shove it into
a tf.train.example. And you serialize
it into a string. And that's it. So this is just converting
blocks of NumPy float data into a serialized format
that TensorFlow can read. Next, please. OK. Now we can put all
that stuff together into an Apache Beam pipeline. And the big advantage
of doing this is that you can send
that pipeline to Dataflow and run it at scale. So that's why I'm showing
you this crazy stuff about serializing
patches and NumPy arrays and so on and so forth
because eventually, you want to write something like
this and send it to Dataflow and run it at scale as an
alternative to exporting those patches. OK. So how does this
Beam pipeline work? First, you just create the
pipeline from those regions. That capital regions is
that list of the polygons that I drew near
Ho Chi Minh there. Then we're going to do a
flat map over those regions. So this is like calling
a map in Earth Engine and then dot flatten
on the result. You're going to flatten out
the multiple things that come from sampling this region. So each region gets sampled. And remember, this is where
you're yielding those patches. Or sorry, you're yielding the
coordinates one at a time. And for each coordinate
that gets yield, you grab the training patch. You serialize it. You split it into two
pieces, write it to TFRecord, and that's it. So this pipeline is
sampling all those polygons, grabbing a patch at each
sample, serializing them, splitting the data
set into two pieces, and writing it to
TFRecord files. OK. Next. OK. Let's see. So I'll call your attention
to our handy little graphic at the top here,
which indicates which stage of the process we're at. We've generated
our training data, and now we're going to
preprocess the training data. So you've written
your Beam pipeline. You ran it with Dataflow. You've got a ton of data
sitting somewhere or other. Now you want to read the data. That's what this thing does. And there's a bunch
of fanciness in here. You interleave the reads. So the reads get
parallelized over the files. You read them into
TFRecord data sets. You parse the data out
of them because remember, we put into those
serialized strings. And now we're going to
parse it back out again. Cache it. Shuffle it. Batch it. And prefetch it. All this stuff, if you go
to the Example notebook, it links to the
TensorFlow docs that explain why you might
want to do all this stuff. But the idea is just to have
a very efficient pipeline for reading the data
sets that you wrote. Next. OK. Now we're going to
define the model. As I mentioned previously, this
is a fully convolutional model. And-- so what are
we looking at here? First of all, I'm not
a deep learning expert, despite the title of this talk. I took a class at Google once. And the teacher
said, grad students have wasted the best years
of their lives generating these models so that
you don't have to. And I really buy
into that philosophy. [LAUGHTER] Your fancy model stuff
goes in the middle there. This is just something
that I grabbed from the Image Segmentation
by Scratch page on the Keras website. That's what the fancy stuff
is in the middle there. Substitute your own fancy stuff. OK. The important point
that I want to make here is about the shape of the input
and the shape of the output. Specifically, the
input shape is going to be height by
width by channels. So note, I've left the
first two dimensions-- the height and
width of the patch-- blank so that you can
use whatever you want at predict time. But the model is trained
on 128 by 128 by 9 patches. And you are expect to
have nine inputs here. So the patch size can vary
within constraints of the model architecture, but it's
going to expect nine bands. So that's the length
of input bands here. So height, width,
channels on the input. On the output, you have
another convolutional layer with the number of classes
in your output as the depth. And what that means
is that you will get a number for each of the
classes in your classification. And that number is the
predicted probability that that's the class that
you're seeing in that pixel. So you will get a 128-- oh, do we have a
quiz on the next one? Can you go to the next one? OK. Now go back up. Sorry. Pop quiz. What is the shape
of the output here? It's kind of a trick question. It's kind of a trick question
because really, the shape is whatever shape you
send in with nine bands. So it's height, width,
nine bands deep. And the output is going
to be the same shape, except the depth is going
to be the number of classes in your classification. So that's what is coming
and going out of this model. And that's what we have to
tell Earth Engine to expect. Next slide. OK. Now you've got your fancy model. You've got all
your training data. It's time to train your model. You can train it
wherever you want. So in this example, it's just
trained in the Colab notebook virtual machine. That works fine if
you've got a small model without too much training data. But if you have a
big, fancy model with lots of training
data, you might want to use some
Cloud infrastructure to do your training. And you can find that in
this second example here. That will train the model on
Vertex AI, which will let you do much longer training
jobs, hyperparameter tuning, lots of fancy stuff like that. Or you can use a
deep learning image, which fires up a
VM for you, which has all the right software
already pre-installed in it. You start up a JupyterLab
server on the deep learning VM, connect to it, and then
do your thing that way. It's like Colab, but you have
control over the size of the VM that you're using. So that's another method. Or you can train it on your
big, fancy university cluster or whatever you've got. Maybe you have a very good
machine that you can just let it run for a while. That's fine, too. At the end of it, what
you're going to have is a trained model. And you're just going to save
it to Saved Model somewhere in Cloud Storage. So you can train it wherever
and however you want. All you have to have at
the end is the saved model. Yes. AUDIENCE: Does that
find the data set? Are you exporting out the data? NICHOLAS CLINTON: Yeah. So the question is,
if you're training on your big, fancy
cluster, you've got to get the data to your
big, fancy cluster somehow. And the way you're probably
going to do that is you're going to stage it in
a Cloud storage bucket and then download it. So yeah, you still have
to get the training data to your machine somehow. Other questions? How are we on time? CREW: You have 20 minutes. NICHOLAS CLINTON: 20 minutes. Plenty of time. OK. This is another new part. Previously, we did
a thing which we liked to refer to
colloquially as EE-ification. And really, it should have been
called AI platform-ification. But that was implemented by
this Earth Engine model Prepare thing. And the purpose of that was
to bolt on some extra nodes to your saved model. So you've trained your model. You saved it. And now you want to send
things back and forth to Vertex AI or AI Platform. And those things that
you send back and forth need to be Base64 encoded. And this Earth Engine
model Prepare command-- the Earth Engine model Prepare
command did that for you. It bolted on those
extra nodes that would do the conversion
from Float32 to Base64. Here's a programmatic
way to do that. So basically, this is extending
tf.keras.layers.Layer. So you're defining a new layer. And inside this new
layer, when you're deserializing your input,
you do decode Base64 and parse the tensor
out of that using the features that you know
are supposed to be in there. And that gets sent
to your model. On the way out, you encode it
to Base64 and serialize it, and that's it. So this stuff,
you'll actually find this in the open source of the
Earth Engine Python client. So you can look
that up on GitHub. You can see the source for
Earth Engine model prepare. That's basically what
this stuff is doing here. But it's doing it the new way-- the TensorFlow 2.x
way, using Keras. In fact, an external
user figured this out. Matt Miller-- shoutout
to Matt Miller. Thank you very much. This is how you do
EE-ification programmatically. OK. Next slide, please. OK. I lied a minute ago. You don't save your model first. You do the EE-ification. Then you save your model. You can also do it
in the reverse order. In fact, that's the second demo. But for now, you define
your fancy model. You wrap it in those extra
layers, and you save it. Next. OK. Now we want to connect
to it from Earth Engine. And the way that
you do that is you host your saved model on AI
Platform, Vertex coming soon. So here we're just
creating a version and pointing it to the
directory in Cloud Storage that has the Saved Model artifacts. So I'm not showing you
creating the model. First, you create the model. Then you create the version. You point it to your code. And next slide. Then you connect to it
using this Earth Engine connector in the client
library ee.model.fromAIp latformPredictor. Now, the important
thing with this is-- well, I tried to bold
the important things. But really, all
this is important. So first, you specify the
project, the model name, and the version name. Then you choose a projection
for your input data. And you set Fix Input
Projection to True. Why do you do this? The reason that
you want to do this is because when you've
sampled your training data, you've chosen a
scale and projection. If you export something
in Earth Engine, you set the scale
and projection. If you don't set that,
who knows what you get? In fact, you'll probably
get EPSG4326 1 degree by 1 degree scale. That's probably
not what you want. We want EPSG4326, which is just
latitude, longitude, in WGS84 at 10 meters resolution. And we fix that
input projection so that when Earth Engine is
sending tiles to AI Platform, it's sampling the tiles
in that projection and at that scale, which
matches your training data. So that's why we do that. That's also why AGV told
you not to zoom out too far, for the same reason
that if you zoom out too far with a reproject,
Earth Engine will try to get every pixel
you can see in the thing and reproject it. So if you zoom out to the
whole world with your model, it's going to try to
sample the entire world and send all those
things to AI Platform. And sadness will ensue,
so don't do that. But you can export this stuff. The other thing is, you
set the input tile size. And recall that I
mentioned previously, these models are
invariant to input size within some constraints. But this is basically
half the input size. But then I'm doing
this other trick, which is specifying the overlap size. So it's going to
overlap 32 pixels on every edge of the patches. And that's great
because sometimes you have edge effects, and you
have to trim those out. So this is a nice way
to sort of fiddle-- you can think of all these
as hyperparameters almost. And you might want to tune
the patch size and the overlap and all that stuff to
get the best accuracy. Any questions about this? Yes. AUDIENCE: Is it possible to
get scale at a [INAUDIBLE] NICHOLAS CLINTON: Yeah. Fascinating question. You would not specify a fixed
input projection in that case. You would let Earth Engine
figure it out for you. AUDIENCE: [INAUDIBLE] NICHOLAS CLINTON: Right. So I think the question
is, can you use scale as an input to your model. Fascinating question. I don't know the answer. There might be
research on that topic. If not, then there's
fertile ground for research right there. OK. Next slide, please. AUDIENCE: Nick, how
long did it take to register on the AI Platform? Because [INAUDIBLE] NICHOLAS CLINTON: Yeah. Can you go back to
the model creation? The previous one-- this. OK. Yes. Thank you. This is a paid service. And when you spin
this thing up, you're going to start
getting billed for it. And so the idea is you spin
it up, do your predictions, and then you turn it off again. But you will get billed for
making those predictions by a hosted model. And furthermore, if
you attach GPUs to it or something like
that, you're going to get an even bigger bill. So it's very slick. But it is a billable service. ALEXANDRINA GV: So
if you remember, training and hosting-- two separate workflows. The reason why
training, for example, in our samples were less than
$5.00 was because we turned it on real quick, and
then we turned it off. And we called it a day. But Vertex AI or AI
platform are services that have a VM that's
on all the time. And now we're talking hosting. So in order for you to get
streaming those predictions, that's where it starts
getting costly because there's a cost per hour. And then there's an
additional cost per hour if you enable an
accelerator called a GPU. NICHOLAS CLINTON: OK. I'm going to move on because
we're running short on time here. This is what you get. This is the input. This is the output. You call Predict to
Image on the input. You get that vector
of probabilities that I mentioned before. And if you want the class
label, you call it Argmax-- array Argmax-- on that image. And then you get the label. And then you get your patch. And this is the result of
my not fancy model at all. So next example-- this is an
image classification model. So patch goes in,
label comes out. This is a very different thing. This is not patch goes
in, and patch comes out. In fact, you're reducing the
resolution of the input here. I sort of struggled to
find suitable training data for this. In fact, I've been doing some
beaver research recently, which I've got this
great presence absence data set for beavers. That's really what you want
for this kind of thing. Patch goes in. What's the thing in there
that you're looking for? If you have that data, great. Here's an example of a
prebaked model that does that. This is the BigEarthNet model. This was a-- well, the
data set is a bunch of Sentinel-2 patches
that were labeled in terms of what's in the patch. And a guy-- a Googler, actually,
out of the Zurich office took that data set, trained
a ResNet 50 with it, and hosted it on TensorFlow Hub. So what this ResNet 50 does
is it takes 120 by 120 patches from Sentinel-2 as input, and
it spits out the list of labels of stuff that you think you're
going to find in there-- airports, beaches, rivers, water
bodies, all this kind of stuff. So that's what this model--
it's just a pretrained model, a pretrained image
classification model. So the problem with this
pretrained image classification model is that the results-- you need to EE-ify this thing. And so here's how that happens. Is that happening here? No, it's not. This is just the model itself. You do have to EE-ify it. This is just the model itself. You go to TensorFlow Hub,
and you grab the URL. You load the saved
model from there. You say, I want to use the
default serving signature to make predictions. And I want to get the logits
out the end of that thing. There are a bunch of
outputs from this model, embedding vectors and
all kinds of fancy stuff that might be useful to you. I just grabbed the
logits, stuck them in the softmax to get
probabilities, and now reshape the outputs. So this is the part that
is to make this model work with Earth Engine. And specifically-- so here
are the probabilities. This is just a length 43 vector. 120 by 120 patch goes in. Length 43 vector comes out. But we need to expand the
dimensions of that thing so that it is-- pop quiz. What shape does it need to be? OK. So in this case, it will be the
length of this vector, which is 43, 1 by 1. So 120 by 120 goes in. 120 by 120 by 3 goes in. 1 by 1 by 43 comes out. And so the real point
that I want to make here is that Earth
Engine is expecting data in this specific
shape so that it knows how to put it on the map. Next slide, please. So you define your
model that way. You EE-ify it. You save it. You host it. And then you connect to it. This connection is a little
bit different, obviously, because it's a
different kind of model. Specifically, I'm fiddling
with the input tile size and the input overlap size here. Just again, these are
almost hyperparameters. I'm not sure what
they should be. I just kind of fiddled
around with this thing. Sometimes Earth Engine will
complain if you don't tell it the shape of the input. And specifically,
what it wants to know is that you're
sending an array in, and it's a length 3 array,
meaning red, green, and blue. That's what the
BigEarthNet model is-- red, green, blue. That's what it expects. And the output tile
size is 1 by 1. So what's happening here is
that 120 by 120 at 10 meters resolution is coming in. 600 meters is coming out. And in each one of those
big 600-meter pixels is the length 43 vector of
all the things that it thinks is in that patch. CREW: Five minutes. NICHOLAS CLINTON:
Next slide, please. OK. So this is what the
output looks like. This thing goes in. Each one of these
is a 60 by 60 patch. You can't see the
overlap because that part gets chopped off. But it thinks that
in this patch here, there's some list of 43
things, some of which might be vanishingly
small probability. So I made this little app which
just shows you the top one. But there's a list
of them in there. Next slide. OK. Am I taking us home? You're taking us home. OK. ALEXANDRINA GV: Do you want
to say anything, though? NICHOLAS CLINTON: Any questions
about these demos before I-- yes, in the back there. AUDIENCE: Is it typical
for the prebuilt model to have a documentation
[INAUDIBLE]?? ALEXANDRINA GV: Depends. NICHOLAS CLINTON: Yeah. Is it typical? I'm not sure. ALEXANDRINA GV: Depends. NICHOLAS CLINTON: This one had
sort of mediocre documentation. ALEXANDRINA GV: The question
was, do these lovely on-demand models have documentation? And the response is,
depends on the author. NICHOLAS CLINTON: Right. AUDIENCE: Is there a way to
estimate cost for predicting a large [INAUDIBLE]? NICHOLAS CLINTON: Yeah. The question is about
how much does it cost. I don't have a terrific
answer for you. But in developing
these demos and running a lot of predictions
over that small area, I racked up about a
$200 bill in a month. ALEXANDRINA GV: For AI Platform. NICHOLAS CLINTON:
For AI Platform. ALEXANDRINA GV: But
if you try to do it where you transform the
NumPy arrays into cogs, it could be different
on Cloud Run. NICHOLAS CLINTON: You
want to take us home? ALEXANDRINA GV: Yeah. So please take a picture of
this, or visit the resources. This first link is
David's end-to-end sample. It covers, defines everything
that Nick just walked through and a lot of canonical
best practices. These are two of
Nick's beautiful demos that he built just for
you for this session. And I think you call them more
simplified specific areas. One's a fully
convolutional network, and the other one is
using an existing model that you pull in. I have my "People on Planet
AI" series on YouTube. We will be launching,
in a couple of weeks, this same content. We're also are going to talk
about weather prediction. Then we're going to do
wildfire prediction. And then we're going to talk
about restoration as well, all through fully
convolutional networks. And then if you want
to stay in touch or stay connected as we
are bringing out content, you can follow me on Twitter
@Open_Eco_Source or the Earth Engine Twitter account as well. And I believe, Nick, you wanted
to say something about this. NICHOLAS CLINTON: Oh, yeah. Yeah. So we did pitch this in the
previous session as well. It would be great
if you would like to take part in a user
survey as we build out this infrastructure. Understanding how
people are using it and how they want to use it
and where the pain points are is really valuable to us. So if you'd like to
participate in the study, please fill out the form,
and someone will be in touch. ALEXANDRINA GV:
Yeah, I'm blown away and impressed at how many
people are in this space. So definitely, if
you are here, make your voice heard in this form. Anything else? Thank you for being here. Have a good day. NICHOLAS CLINTON: Thank you. [APPLAUSE] [MUSIC PLAYING]
