[MUSIC PLAYING] NATALIE VAIS: We're going to be
talking about Firestore today. My name is Natalie Vais,
and I'm a product manager on Cloud Firestore. I will be joined by my
colleague Ben Whitehead who's on our developer relations
team, and he will come on stage later on in the presentation. So first I wanted to start
with a quick overview of the session. So the main goal
of this session is for you to walk
away understanding what Firestore is,
what it's good for, and how you can integrate it
with other parts of the GCP ecosystem. For those of you who
are new to the product, I'm going to start
with an introduction and a history of the product. I'll highlight some of the
new product announcements that we made this
year as part of going generally available in January. I'll walk through some
of our example customer architectures of people that are
using this in production today. And then finally,
I'll give it over to Ben who will walk you through
a demo demonstrating Firestore in action. We'll also leave some time at
the end of the session for Q&A. And speaking of Q&A,
how we're actually going to do it in
the session today is using an application
called Dorie which we use internally at
Google to manage Q&A at scale. You can actually
find this through your Next mobile application
for this session. This will allow you
to submit questions throughout the period
of this session. And you can upvote and
downvote other peoples, and I'll be using
that to moderate at the end of the session. So please feel free
to check that out. So let's go ahead and dive into
an introduction to Firestore. Before we jump
into that, I'd like to get a bit of a sense of
the audience we have today. How many of you out
there have actually used Firestone already? That's awesome. Looks like 40%. How about the real
time database? Firebase Realtime Database. OK. Less of you. And how about Cloud Datastore? Cool. Sounds good. That's good background
for the session. So Cloud Firestore is one
of many database offerings that we have within
Google's cloud portfolio. Here you can see a list of
all of the managed database services. Cloud Firestore is our NoSQL
document-oriented database, making it part of our
non-relational offering. And we'll talk about this
data model a little bit later in the presentation. I want to take a
step back briefly and imagine that
you're trying to build a mobile app for your company. If you're starting a
project from scratch and you want to
build it yourself, you're likely going to have to
think about a couple of things. Number one, setting
up a database, which you'll probably have
to run and scale yourself if you're taking
something off the shelf. In front of that, you'll need
to put some sort of API layer that can make queries
from your clients. That probably means also
adding an authentication layer so that you can make sure
that users that are accessing your data are authenticated. And this also means adding
some sort of sync logic so that you can make sure you
make sure your data is syncing with the local results. And finally, these local
results are probably going to be stored in a
completely separate client database that again,
you have to make sure is syncing
with your database that we mentioned
up at the front. Now when you arrive
here, all of a sudden you have a pretty complex
set of solutions. Each of these as a
standalone could actually be a whole company in itself. And that's what we
actually see today is that people are having
to go to many vendors, many different places, to
actually accomplish this stack to build their
mobile or web apps. And this is why we
built Firestore. Firestore was built to
tackle these problems that are very common to mobile
and web app development and to simplify this
process for you. For those that are familiar
with Cloud Datastore and the Firebase
Realtime Database, an important question is, how
does this relate to those? Cloud Firestore is
the next generation of cloud data storage. And it includes
important learnings from the Firebase
Realtime Database. So we took a look at
the developer wishlists from each of these,
and we used those to inform the
decisions that we made when built in Cloud Firestore. So you can think of this
as the best of both worlds. It takes some of the best
elements of Datastore and some of the best
elements and learnings from Firebase Realtime Database. The first thing to know
is that Firestore is fully supported in the Firebase SDKs. This includes web,
iOS, and Android, so you can access data
directly from the clients. This is actually pretty
unusual for a database, and it's not lost on us that
letting clients talk directly to your database can pose
all sorts of challenges. For one, you can't
always trust the data that's coming from clients. And secondly, you're dealing
with potentially unauthorized users. So this is where Firebase
Auth and security rules come in handy. These allow you to validate
operations from the client, and this removes the need to
build that middle tier that you saw on the previous
slide, which can sometimes be its own standalone server
just to manage authentication. Now the next thing
that's very important is the ability to have
these applications work across multiple
devices when we're talking about client access. Maybe you put down your
phone, you walk over to your desktop
computer, and you expect to resume the
same state in your app that you're operating on. So this is something that
Firestore handles natively, and it makes it really, really
great for those applications that you want to
develop and serve to multiple types of
devices at one time. Now another problem
that's very, very common these days is people expect
their apps to work offline. It's fairly normal
for people to look at their mobile
applications when there's little to no
chance of connectivity. Maybe you're running
through the airport, you're at the top of
a mountain, or maybe you've gone offline on purpose. So the same
technology that allows us to operate in offline mode
also helps in the situations where we have spotty
network connection. And so as you can imagine, this
is very challenging to solve. And we've actually built
this natively into Firestore. It requires building complex
sync and conflict resolution logic on the client. So in Cloud Firestore,
our client libraries come fully equipped to
handle these use cases. So let's talk about the
data model in Firestore. We've mentioned already that
Firestore is a document store. So at the smallest
unit, the smallest unit of storage in Firestore
is a document. And a document, as you can
see up here on the screen, is a lightweight record that
contains key value pairs. There is no schema
enforcement in Firestore, so you have complete
freedom over what fields you put in a document. Now documents can be grouped
together in related groups that we call collections. And you can see that
representation up here. Documents can also map
to sub-collections, and this lends itself really
nicely to hierarchical data that's very common in web
and mobile app development. So what we've seen so
far is that this model works really, really
well for developers, and is designed really
well to scale as well. So let's talk about what's
new in Cloud Firestore. As we mentioned,
called Firestore went generally available
just earlier this year at the end of January. It was in beta for a little
over a year before that-- since 2017. So as part of the announcement
of going generally available, we have some pretty cool new
things that we've announced. So the first thing
I want to mention is that Cloud Firestore
operates in two different modes, Datastore Mode and Native Mode. These have slightly
different strengths that I will highlight
based on the use case that you're looking for. So the first thing is
that Datastore Mode has the exact same API and
same scale as Datastore, but a new storage layer. So if you are a
Datastore customer, this will feel very
familiar to you. And in fact, eventually, all
existing Datastore customers will be able to upgrade to
Firestore and Datastore mode. Datastore mode also
has some new things that are greater than
some of the things offered in Datastore today. And those things include
strong consistency, where we had eventually
consistent queries before in Datastore. We also have a higher
availability SLA, which I'll talk about in a second. And we also have a new
data flow connector. So on the other side of
things, we have Native Mode. And Native Mode leverages
this new document model, and it integrates with
third party clients. And that's kind of
the main difference. And for those of you coming from
the Firebase Realtime Database world, it's more
akin to the model there with the
direct line access. A simple way to
think about this is that Datastore Mode is
optimized and scales really, really well for your rights,
and Native Mode scales really, really well for
concurrent connections. So when you're thinking about
all these different clients trying to connect to the
same application at once. As part of going
generally available, we also announced
10 new regions-- 10 new locations that
are a combination of multi-region and regional. So you can see we have
two new regional-- sorry, two new
multi-regional locations that are highlighted in green. We have one in the
US and one in Europe. And these multi-region
locations can actually withstand the loss
of entire regions because they're replicated
across more than one. And this maintains availability
without losing data, and all of these regions
are actually required to be at least 100 miles apart. In addition, we have new
service level agreements that are attached to each
of these different types of locations. So the first for multi-region,
which is our highest SLA-- we have five nines availability. And for regional, we
have four nines SLA. Both of these are actually
improvements over the SLAs that you saw in Datastore, which
were four nines and three nines respectively. And lastly, you can
access Firestore directly from the Cloud Console. Many of you who come from
the Realtime Database world may know that you can
also access Firestore from the Firebase Console. And that's very common as well. One of the things to
note about these UIs is that they both operate
on the same projects. So if you create a project
in the Cloud Console, it's the same
project that you'll see available in the
Firebase Console as well. One of the benefits of using
the Google Cloud Console is that you have access
to all the other GCB products that I mentioned,
including things like BigQuery. And that's something
that we're actually going to showcase
later in the demo is how to get some
of the data that's in Firestore into BigQuery. So let's walk through a
couple of customer examples. Since we went GA, we've seen
a huge diversity of workloads since launch, and some
of these were actually some of our early beta customers
as well back from 2017. So we have a lot
of customers that span tons of
different industries, but they do have a couple
of things in common. The first is that a
lot of these companies needed a really
quick time to market. So they didn't want
to spend all this time provisioning servers,
picking things off the shelf, and stacking things together. Second is that they
needed something that would have the ability
to scale with their business. They wanted the same
solution for their 10 users that they want for
their 100,000, 1 million user use case. And the third thing is that
a lot of these companies had really lean
development teams, and they wanted to operate
in a serverless environment. And as we know, Firestore
is a serverless environment. You don't have to
think in terms of nodes that you're provisioning or
capacity and resource planning. In fact, the pricing is actually
built on an operational model based on the reason rights
that you're actually doing against the systems. And this is huge, because
Firestore does not require a massive team to run it. And that's the beauty of it. The same team that
gets you started can actually carry you
through the scaling of your application. The first one of these customers
that I want to highlight is a company called
Hawkin Dynamics. They build sophisticated
hardware and software for athletic
monitoring that's used by several professional
sports teams. So this hardware
that they develop includes what you
can see up here on the screen, which is
wireless force plates. And these wireless force plates
measure the impact and movement of athletes, and they
can track and measure the performance
of these athletes over time by capturing
this isometric data in their application and
sending it back too Firestore. Here's what their
architecture looks like. They have the
wireless force plates which capture and send the
data to the Hawkin Dynamics mobile app. This collects information across
athletes in many locations. And because they're testing
facilities are often in low-connectivity areas,
as you can imagine-- fields, maybe they're doing
something at high altitude-- the ability to have
this offline sync is very, very important to them. So they can do their
tests, capture the data for the athletes, have it
sent back to the application, and then they have coaches
or other people that are monitoring the athletic
statistics connecting from other devices like
tablets and desktop computers that can access this
information immediately. So this is very
important to them. And they've actually
been kind enough to share their data models. So we can drill into a little
bit of how they've actually modeled their business using the
Cloud Firestore document model. So what you'll see up is a top
level collection called teams. Each of this collection
contains documents, one of each corresponding
to each of their teams. And these teams have
sub-collections. And these
sub-collections, you can see some of them listed here. It can include
settings, raw force data that's collected from the
force plates, different tests. If we drill in a level further,
they also a sub-collection called Athletes. So here there's a document
for every one of the athletes that they're collecting
this information on. And here they can drill
into all the tests that have been
captured or performed for that athlete over time. The next customer that
I want to highlight is one that you actually
may be familiar with. There's a high chance that
you went to this website because you're all
here today, or that you accessed the mobile app. So our developer friends
over at the Nerdery-- I think we have some of them
potentially in the audience. There. So they are a
digital consultancy that actually helped build the
conference web and mobile apps for us using Cloud Firestore. So they use Firestore
as an aggregation hub for all of the
conference data that's serving the session information,
the breakouts, et cetera. And again, we actually
have a data model to show you and
see how this works. So here you can see
an actual document that represents content that's
being served to the next home page. And in this case, it's
the mobile app page. So you can see the
content being served here. It also has static content
that's being stored elsewhere. And you can see the path
to that information here. If I click to the
next, this is actually a high level collection
representing all of the different next event. So the one that's
highlighted right now is the one that you're in here
today for, which is Next SF. You'll also see entries here
for the future next event that we have in
Tokyo and London. And here is actually the
session we're in right now. So this document
represents the session ID. You can see here
we have information about the session type, the
different related fields-- so these are sessions
it recommends you go to if you also
have gone to this one-- and then speaker ID, which
refer to both Ben and myself. Something that's not on
the screen that also exists in the app is a
field lower down that tells the capacity of
each of the sessions. So you can imagine if
the capacity is full, it will no longer let
someone register for that. So when you first visit the
conference site, what you're seeing is a session
and speaker information that's being loaded
from Firestore, as well as some of these static
images and assets from Google Cloud Storage. When you sign in with
either your Google account or personal email
address, you're actually hitting
Firebase Auth which is authenticating you
and making sure you have the proper account and
you have access to all the information you're seeing. Then when you register for
a session like this one that you're in today, the
app makes an API call out, it updates the
information in Firestore, and immediately updates that
information to the user. So next what I want to
do is invite Ben onstage. Now that you've seen
some customer examples, I want him to
actually demonstrate Firestore for you in action. Thank you, Natalie. [APPLAUSE] BEN WHITEHEAD: I
am Ben Whitehead. I'm a developer
programs engineer with Google Cloud focusing
on Cloud Firestore and Cloud Datastore. So I'm going to be
running through a demo today showing how we've got
a demo app built leveraging the real time functionality
of Cloud Firestore, as well as some of the
larger suite of GCP to handle doing analytics
tasks to kind of help the business understand
how your app is behaving. So first, we're going to look
at a couple of architecture diagrams just to give
a high level view, and then we'll dig into
some of the details. So here we've got a high level
architecture diagram of the app that we're going to be
demoing and analyzing today called Firepixels. In the little mobile
icon, you can kind of see that there are
four colored quadrants. This is actually the
Firestore logo colored using the Google colors. So each of these
respective quadrants is made up of 10,000 pixels. And each of those
10,000 pixels are backed by a document in Firestore. So in Firestore, we
actually have a document for every single pixel
that's maintaining the color of that pixel,
the last updated timestamp, as well as a
partition that we're using to parallelize the
event propagating to the UI that we'll be able to see in
the real-time demo in just a few minutes. So given that there are the
four quadrants, each of them have 10,000, this can handle
up to 40,000 concurrent users, essentially kind of
poking at the bits to represent each of the pixels. So we've got a
simulation service running on GKE that's
actually running a simulation of
some user activity, so you will be seeing
pixels kind of updating when we get over to the demo. So we've got this
application up and running. We've gained a fair
amount of user traction. And the business comes
over to us and says, great. You've got this thing. We need to start
getting some numbers. And because you were
very quick to market, that probably didn't mean that
you baked all of your analytics into your app from the get-go. So how do you reinforce and
get information for your users? So the particular
business questions that we're going to
be answering today are, what are the number
of distinct color events that have happened over
the past 30 minutes, over the past 2 days,
and over the past 7 days? The next question is,
what are the number of distinct colors that
occurred in the last minute for the past 30 minutes? And finally, we're
going to have a chart that we are going to use
to monitor the ingestion latency of how long it's
actually taking to process these events from user
actions to it being available in the data warehouse. So this is a high level
architecture diagram essentially expanding
on what components we're going to be looking at today. So on the left hand
side, the yellow box-- it's a little faint. Sorry about that-- the left hand
box represents the same side that we just saw previously. So we can see here,
Cloud Firestore is sort of the bridging
point into the rest of GCP that we're going to be
integrating with today. And we're going
to specifically be using Cloud Functions to
monitor change events on each of the pixel documents
inside the database. When each of those
events is triggered, we're going to
handle that event, process it, and feed that
into BigQuery that will then be used to generate some
predefined views to feed charts inside Data Studio
so that we can put up a dashboard in our office for
everybody to able to look at. And since Cloud
Functions are these kind of magic, ephemeral
things, we need to have some monitoring for us. So we're going to
leverage Stackdriver to do essentially
metrics monitoring and invocations per second,
and those sorts of things over time. Can we switch to the
demo laptop, please? OK, great. Over here on the left hand side,
this is the Firepixels app. This is a real time app. This is running,
hitting Firestore. We can see each of the four
colored quadrants currently of-- oh, look. Cool. Our simulation just kicked in. That bottom left
quadrant was updated. And we've got a number
of document listeners on the back end that are
listening for those simulation events. So that was just 10,000 writes
to the database and the events being triggered, audited,
sent to the back end, as well as aggregated
and then sent down to the web client
over web sockets. So it's very responsive. It can give you a great
experience for your users. And we'll be able to see
that changing over time during the rest of
the presentation. So first, we're
going to jump over to Cloud Firestore in
the Google Console, and we're going to take a look
into the data model that's actually backing the Firepixels
app right now on the left hand side. So here on the left
hand column, we've got three different collections,
top level collections. One for devices, one for
input, and one for pixels. The collection for devices-- this is actually a
heartbeat collection. So for each of the
respective clients that are managing each
of those pixels as part of the simulation,
we're essentially tracking the heartbeat of the
last time one of those users actually did something for us. If we move down to
the inputs collection, we can see a 0 through 3. The 0 through 3 here represent
each of the colored quadrants over on the left hand side,
starting top to bottom, left to right. Excuse me. Left to right, top to bottom. So 0, 1, 2, 3. In each of those documents,
we can see the color. We can also see the updated
timestamp of the last time that quadrant was updated. And if we look at the
pixels collection, here we can see a list
of the 40,000 documents that represent
each of the pixels. So the name of the document
that we've got here is essentially 3
digits dash 3 digits. This is representing the
xy coordinate in the grid. And we can see the color
for that particular pixel, which partition that
pixel is associated with, and the last updated timestamp. So while the
simulation is running-- wow, we got it all blue. That's really unusual--
let's take the top right and let's change
this to be yellow. So I can hit that. That's going to
trigger the events. All of the event
handlers on the backside are going to observe
that particular part of the simulation. They're then going
to update each of their respective
pixel documents. And then, again, that's going
to be aggregated to the back end and sent over to the UI. So you can kind of see
how quick and responsive that is, even from an
administrative standpoint. From there, let's jump back over
to the Google Cloud Console. A quick overview here-- the Google Cloud
Console is trying to help you out and
surface information that's relevant to you based on
how your project is behaving. So for our project, we are using
Compute Engine to run our GKE cluster for our simulations. We've got some GCS buckets,
Google Cloud Storage buckets for storing some static assets. We can also see that we're
using Cloud Functions. We currently have one
function that's deployed. And we've got one data set in
BigQuery, which is ultimately the data warehouse
data set that we're going to be working with today. And we can also see a quick
overview of the number API requests, as well as compute
engine CPU processing. The API requests--
this is aggregated across all of the various
back end products in GCP. So this can give you a good
idea of how much of GCP you're using without
necessarily having to build up a big graph all yourself. So for here, let's start
from Cloud Functions. So I'm going to open up
a new tab real quick. This is the single
Cloud Function that we have active right now. If you had more
Cloud Functions, they would show up in this list. You get a quick
overview of what region the function is running in. This is important
for data locality just in terms of
latency of processing. It's also becoming
increasingly more important due to regulatory
restrictions around data egress and migrations,
and those sorts of things. The trigger type
of the function. So there are many types
of triggers that you can use with Cloud Functions. Today, we're using
Cloud Firestore. You can also use it with
Google Cloud Storage when a new document is
created or other sorts of things like that. We're, for this example,
we're using Node.js version 8. This is convenient. It gives us the
async await handles. So it makes our code a little
bit more terse, which is nice. I've given this function
blindly 256 megabytes of RAM. I seriously doubt I'm using
that much, so I should probably knock it down to
save on some billing. But, you know, I haven't
been harassed about that yet, so we'll worry about it later. And then the function
inside our code that's going to be called
as part of the event handler is called On Pixel Change. And we can see the last time
that this code was deployed. So let's click in here and
load up some of the details. When we click into
each of the functions, we get to a high
level overview showing the number of
invocations per seconds where we can kind of see
success, failure, error rates-- those sorts of things,
just in high level view. This little peak that we notice
on the right hand side, this is when I manually change
the value in the database a couple of minutes ago. Those events got
processed immediately. So that's higher than the normal
simulation is normally doing. You can see that you can
get some different time granularities. But ultimately,
this information is bounded to what time
boundaries you have here. So for longer term storage
and monitoring and alerting, we'll leverage
Stackdriver for that and we'll get to that
in just a minute. So scrolling down a
little bit, one thing that's not shown
on the summary page is this service
account down here. The service account is
actually very important. So the service account gives
your function an identity as far as access
controls and security are concerned in the greater GCP. So when I'm integrating
with BigQuery, the identity of next IO
storage service account is given the permission
to write into BigQuery. So if you've got many
projects or many teams, you can do fine-grain controls
to where you could say, this function is able to write
to this thing in BigQuery, but this other function is
only able to interact with GCS. So it gives you
that kind of control to be able to handle multiple
things at the same time. So jumping over
here to the trigger, we saw the trigger type
earlier is Cloud Firestore. The particular event type
that we're caring about right now is update. There is also the Create
and Delete, as well as just general Right. So this can be really nice
from an auditing standpoint, or doing data ingestion
for analytics, as we're doing today. Without having to do
modifications to your database or to your app,
you can essentially hook up monitoring and
those sorts of things. And then the resource
path right here. So if you remember
back here in Firestore, we've got our pixels collection. If you think of this as sort of
like a directory structure that matches with the
document collection, sub-collection document
sort of pattern, right now, essentially, I'm specifying
a directory structure that says underneath the
pixels collection, I want to match on any pixel. So inside curly
brace is essentially like a variable placeholder. So over here on the Source tab-- I'll pull this up for folks. So this massive 30 lines of
code is the sum total of code that you need to be able to
get an event from the Cloud Function into BigQuery. So the BigQuery
node module-- we're able to just import up here the
data set that we're inserting to, the table that
we're inserting to, and we're creating the BigQuery
client right here on line 5. You'll notice there's
no credential passing or, hey, load up this thing,
or anything like that. The default constructor
for the clients is defaulted to using
the service account. So if you're running
something on Cloud Functions or you're running something
on App Engine or GCP, the environment is essentially
bootstrapped for you already to use the service
account information that you've configured
through the cloud API. So moving down a little
bit, lines 7 through 26 are the full function. At line 8 and 9,
we're essentially pulling out the time 0 values. And then on line 9, we're
pulling out the time 1 values. And then we're constructing
our data structure that we want to
send to BigQuery. So that's the pixel
that's being modified-- essentially the pre and post
values, as well as the process time when this is actually being
processed and sent to BigQuery. Then on line 21 is where
we're doing the actual write to BigQuery, and our
function is effectively done. We do have a console
log statement here. I should have cleaned this up. It's not actually doing anything
useful for us at this point. But the logs that I
write to the console here are actually available
in Stackdriver. So if you're setting up your
function and you're like, hey, did I actually
get the data there, or I'm kind of curious
what the events look like, that sort of stuff, you
can do your printf, console log inspection, and all that
information shows up in Stackdriver. OK. So jumping back to the
Cloud Console real quick. From here-- pardon me. I actually skipped Stackdriver. So let's pull up
Stackdriver first. So Stackdriver, if
folks aren't familiar, is the kind of user-side
monitoring hub for your apps and usage of GCP on GCP. So here, we can see
essentially a metric graph. It looks very similar to
the overview-- excuse me-- the overview that we had product
back on the function page. But it does give you the ability
to do broader time spans. So we can actually look back
and see the development work that picked up in
the middle of the May for this particular demo. You can even do custom
frames if you wanted to, as well as real-time
dashboards if this is something that you want to put
in your team's bullpen or something like that. So let's jump back
down to an hour. So right now, we've got
three different charts. So we're tracking the number
of executions of our Cloud Function per second. In the case of Stackdriver,
this is being aggregated up to per-minute buckets. So we can see over time. Again, that little peak when I
manually changed the database. Scrolling down, we can see the
Cloud Function instance count. So this is one of
the things that's really powerful about Cloud
Functions is it essentially takes care of all of the
scaling factors for you. So by us triggering
events and events being in cued to be processed,
GCP figures out how many functions it
needs to scale up for you, and when they can be able
to scale down, and back up, and those sorts of things. So that's why you're
seeing some spikiness this is when it's scaling down,
it's trying to save me money. When it scaling back up, I'm
giving it more work to do. The final chart
down at the bottom is a pretty cool little chart
that Stackdriver provides which gives you a heat map. So instead of having to look
at you're 7 lines for your P50 and P95 and all
that stuff, it'll actually just render
a heat graph for you. So we can see here
that most of our events are processing in
about 100 milliseconds. This little hot spot down
here when I did the manual trigger again. But this is nice,
because you can start to see things over time. If we were to expand
this to the last week, we can see that it's been
running relatively consistent. We've had a couple
high latency pokes at a couple different
point in times. But nothing super concerning,
as they've come back down. And it looks like our peaks
are about 5 seconds or so. So jumping back to
the Cloud Console, from here we're going
to jump into BigQuery and take a look at the data set. So this is the schema
of the pixel tables. You notice it looks exactly
like the JSON document we looked at in the function. Unlike Firestore,
BigQuery is schema'd. But this is to help out
with query planning and just the dynamic nature in
analytics, and the fact that it's SQL-oriented. So we click on Preview. We can actually see a
summary of some of the data that's in there. So for pixel 66, comma, 103,
we changed from red to blue, partition stayed
the same, and then we got the respective
timestamps. This all supports your standard
SQL that you're familiar with. So I could do select star
from the Cloud Console. Also has nice auto
completion for you, which is really great,
because inevitably you end up getting long names,
and historical names, and all that stuff. Let's see. Order by. T1 updated. Descending Limit 10. And we can hit Go, and
it will essentially just start crunching. So in our full
database so far, we've got something like 600
million events that have been stuck in our data warehouse. And that's in the span
of about two weeks. So as apps get more
IoT and user-intensive, you can imagine
having a database that can scale out and give you quick
queries is really valuable. And that's what we're
getting from BigQuery. So we're going to jump down here
and look at a view real quick. So the views that
we've got defined here are essentially what we're using
to drive the charts that I'll show you in just a second. You can see that they're
kind of standard SQL. Select, From, Table,
Group By, Ordering, Limits, all that
same, normal stuff. So you're not having to
learn a brand new charting library or anything like that. You can just write
your SQL statements and then load that
up in your dashboard. So I'll jump over here to
the Firepixels dashboard. So this is a dashboard that
I've created in Data Studio. One of the things that's
really nice about Data Studio dashboards is it fully
integrates with all of your G Suite organization structure. So this dashboard that
I'm showing you right now is actually shared to my demo
user from my main work user. So this is really
nice if you need to put it on a display
in your bullpen, or if you only want to share
it with maybe your manager before it's published to
the rest of the company, you've got full ACL control
the same way that you do with Google Docs. So here on the top row, we
can see three different charts with color breakdowns. So these color breakdowns
are answering the questions that we mentioned at the
beginning of the demo around the business questions. So the business question
is essentially, how many-- or, excuse me--
for the top row is, what is the ratio of a
particular color per time bucket over a period of time? So here we're saying for the
last 30 minutes, about 63% of the color changes were to
blue, and the other 37% or so were to yellow. So this makes it really
easy to look at the chart. And having some context around
the way that the app behaves, you can just kind of
look at it and get an idea of what the
app probably looked like at that point in time. And then that expands
to two days ago, as well as over the past week. In the bottom left
we've got our line chart that's showing the
number of distinct colors that were changed
to in that frame. So here we can see that we
had two that we changed to. One time we only had one. One time we actually got
all the way up to three. You know, just various
interesting things. And then the center chart is
showing our ingestion latency. So this is essentially
showing in seconds how long it's taking
from the document being written into Firestore for
us to write it into BigQuery. So this is
essentially showing us an end-to-end latency of getting
data into our data warehouse. So the vast majority are being
handled in less than 2 seconds. And we've got a little
bit of plateau here. This is actually I
believe due to the fact that the Cloud
Functions are spinning back up after being
spun down between stages of the simulation running. And then there's almost a
full drop off by the time we reach 7 seconds. So the vast, vast majority
of all of our events are being put into our data
warehouse in 8 seconds or so. [MUSIC PLAYING]
