SPEAKER: So if you've
been following along with this video series, maybe
you've noticed something here. Cloud Firestore
consists of a bunch of documents,
which are basically key value pairs, as well
as collections, which are collections of documents. But then, within
these documents, we let you store maps,
the JSON-y looking things, that also kind of work
as key value stores-- you know, like a document. Not only that, but
you can also store arrays, which looks
suspiciously like collections. So at some point, you might
start asking yourself, well, why do we even have
separate documents when we could just, say, put
all of our data into these maps? Why create a subcollection
when I could just create an array of maps,
or even a map of maps? When is it appropriate to
use one versus the other, and what would work
best in my app? Good questions. Let's see if we can find some
answers on this episode of "Get to Know Cloud Firestore." [MUSIC PLAYING] So when it comes to
determining the best way to structure your
data in Cloud Firestore, it's helpful to
remember some rules about how Firestore works. Let's go over some of these now. All right. Rule number one,
documents have limits. So there are some limits when
it comes to how much data you can put into a document. For starters, you're
limited to one meg total of data in a single document. And I know that, in
our world of, like, high-res, 16-megapixel
photos, one meg seems tiny. All right. Cheese. Whew. Oh, you got to
turn the flash off. But when we're talking
about text to numbers here, and we generally
are, that's still a lot of data you can fit
into a single document. For example, the complete text
of "A Tale of Two Cities," 300-page book from an author
who is paid by the word, only about 770k. You could fit all of that
into a single document, and still have room for a good
"Sherlock Holmes" mystery. Also, another rule
around documents-- and this will come
into play later-- is that they can't have more
than roughly 20,000 fields. And by the way, this also
includes fields within the maps that you have in your documents. So this is six fields,
and this counts as seven. One for the overall
address field, as well as one for each individual
element in that address map. And this would count as nine. And one big reason
for this limit is that Firestore
is creating indexes for any field you have here
in your document-- even, yes, inside a map. Basically, from
Firestore's point of view, my document here has a
field called address.city, and it's going to index it. And even if you don't hit
this 20,000 field limit, having documents with
a ton of fields in them can impact your
write performance. Because Cloud
Firestore needs to redo every single one
of these indexes when you create
another document, make a change, and so on. One other rule
around documents is that you're generally limited
to one write per second on the same document. So if you've got a situation
where, like, a lot of clients are all trying to write to
the same document all at once, you might start to see
some of these writes fail. Now, your client libraries
are generally smart enough to retry these writes with
some incremental back-off if that starts
happening, but it's still a situation you want to avoid. On the other hand,
writing to lots of different documents
in the same collection, that's generally fine. Go crazy. OK. Let's move on to
rule number two. You can't retrieve a partial
document, at least not using the client SDKs. So just because you can
put "A Tale of Two Cities" into a document doesn't
mean you should. See, when Cloud Firestore
retrieves a document, you're retrieving all the
data in that document. You can't retrieve partial
data from a document using the client SDKs. So imagine I have my collection
of Charles Dickens books, and my documents
have a title field and a "the entire contents
of the book" field. Then I decide, hey,
you know, maybe I just want my app to show a
list of what Charles Dickens books are in my collection. So I query this collection. Well, hey, guess what? In addition to
grabbing the titles, I'm also grabbing
the entire content of all the stories,
which means I'm now downloading approximately 38,000
times more data than I actually need. And that's bad, right? This is something your
user is going to notice. This is going to affect your
app's performance, its data usage, and its
battery consumption. And this tends to put
your app in a state I like to call getting uninstalled. Sorry, buddy. I can't keep it around. It's killing my data plan. This also has implications when
it comes to security rules. You can't create a
security rule that gives you access to some parts
of a document and not others. If you want to keep part
of a document secret, you'll need to place
those secret parts into a separate document. And I'll talk more about that
in our video on security rules. OK. So now you're thinking, well,
gosh, clearly the answer is to break my data up
into as many tiny documents and as many tiny
subcollections as possible. Well, hang on there. Because we have rule number
three, queries are shallow. So you probably
already know this, but when you grab a document,
you don't grab any of the data in any of these subcollections. And this is generally
a good thing, right? It means you can store
your data hierarchically, in a way that feels
logical, without having to worry about grabbing
more data than you intended. So if I took that content from
my "entire text of the books" field, and moved it down
into a chapter subcollection, well, now, I can grab my
list of Dickens books titles without having to grab all
the text associated with them. However, there's a
flip side to this, which is if you think
you're almost always going to be showing data together,
it might not make sense to put it into a subcollection. For example, maybe I decide to
put my list of main characters into a character subcollection. In many applications,
this might make sense. But if, in my app, it
turns out that whenever I'm showing the
title of a book, I'm also going to be showing a
list of all the characters, well, now I can't grab all that
data I need in a single query. I now have to get
the book and then get a list of all the characters
that are in that book by querying each
book subcollection. And this is going to make
populating a list like this a lot more complicated. So if this ends up being
the look and feel of my app, well, maybe I don't want to put
this data into a subcollection. I can keep it as part
of my main document. There's also another reason
this setup may be bad, which is rule number four. You're billed by the number of
reads and writes you perform. The other disadvantage of
putting these characters in a subcollection
like this is that-- and again, this
is only assuming I want to grab the documents
in this subcollection every single time I grab
this main document-- I've increased my reads here,
from one read to several, depending on how many
characters I want to read in. Now, how much this
matters is probably a factor of how often you'll
be making queries like this. I mean, if this structure
means your app is going to be making 2 million
extra requests per day, well, you know, that's
only roughly $0.12 a day. On the other hand,
if your app is making 2 billion extra requests
because of this structure, well, then that's $120 a
day and probably something worth optimizing. Like I said in the last
video, I'm not always a fan of premature optimization
for billing purposes, particularly if it
comes at the expense of more complicated
code, but it's definitely something to be aware of. OK. Rule number five,
queries can only be used to search for
specific documents within one collection. And they do so by looking
for specific fields that match a certain criteria. Hang on, Todd from 2018 Todd from 2019 is gonna
cover this rule. Lemme take it from here.
So on this side, I have my Charles Dickens characters
stored as a subcollection for each book.
I've included their name and an occupation as two
different fields. On the other side, I have the same list of characters stored as a map field
directly inside the book document, with maybe the character
name as the key and their occupation
as the value. Now, these two structures let
me run very different types of queries. With my characters
in a subcollection, I can say, hey, show
me every character in "Great Expectations" with
a name that starts with p. Or every character in "Oliver
Twist" who's a doctor. And if I wanted to say, "Hey,
show me all characters in all books named Oliver," I can now do that with
a Collection Group Query Now, this would require that I set up a
Collection Group index for the name field in the Firebase Console, and
do keep in mind you're limited to about 200 of these. On the other hand, what if I
put characters in a map like this? Well, I could kinda say, "Hey, show me
every book that has a character whose name is Oliver." And I would do that by
looking for a field called characters.Oliver with a
value that's, like, greater than an empty string. But I'll be honest, this is kinda weird
in that I'm relying on there being a field in a map
with a key named in a particular way, and that seems all sorts of error prone to me. In addition, I wouldn't just
be returning character data here. I'd be returning information
about the entire book that contains this character, which
may or may not be what I want. And I wouldn't be
able to do queries like, show me every character whose
occupation is socialite or show me all characters sorted by name. So my search options are a lot more limited
here. Yeah, I'm pretty sure socialite is an
occupation. It's just that in 2019, we like to call them
"Social media influencers." Now a third option is to put
your characters in a top-level collection. This was a nice workaround before
Collection Group Queries were supported, because they
let me search for, say, characters by name, or all characters
in a particular book, and so that search across all characters
in all books. And if you'll recall
from our earlier video, queries in Cloud
Firestore are very fast. So this "Show all characters
in Oliver Twist" query in the top-level collection
really isn't any slower than grabbing all
the documents from a subcollection. I think the biggest
drawback here is that if I wanted to, say,
search for all characters in Oliver Twist sorted by name,
well, now that requires a composite index because it
is now a multi-field search and, you'll recall, I'm limited
to 200 of those. So if you think you're gonna
be searching for individual records from a group of data,
you should put them into a collection. And you can use
either a sub-collection like this, or a completely separate
top-level collection. As a general guideline, I'd
probably say, store data hierarchically if you
think you're mostly going to be searching for
items per subcollection and only occasionally
want to do a collection group query. And put 'em into a top-level
collection if you think you're mostly going to be searching
across all documents and only occasionally want
to do, say, a per-book query. And putting your data into a map
like this, it's not really useful from a querying standpoint, This would more likely be
useful if you're just storing data you know you're always
going to want to retrieve with the parent document. All right, back to you, Todd
from 2018! Whoa... too far! (Grunts) This might make
more sense when we get into some more
examples in the next video, so stick with me
if you're confused. We have one more
rule to get through, and that's rule number six,
arrays are kind of weird. So there's a really
good blog post out there on why arrays are evil,
but basically, a lot of traditional array
operations don't work very well in a service like
Cloud Firestore. Because when you've
got data that could be altered by
multiple clients, it's very easy to
get confused as to what edits are
happening to what field. At least in a map, if I have
several clients all trying to edit different fields, or
even the exact same field, I generally know what's
going to happen, right? I'll end up with a
map that has fields that are equal to that of the
most recent client update. But in an array,
if one client is trying to edit a
value at index two, and another client tries to
delete the value at index two, and another client tries
to insert a value somewhere at index zero, I'm going to
have very different results, and possibly out
of bounds errors, depending on what order these
instructions are received in. So for that reason,
Firestore's actions with arrays are a bit different than
what you might expect. For instance, you can't perform
actions like Insert or Modify, or delete an element of an
array at a specific position. And you can't run a query
on the element contained at position 2 in the array. So are they useless? Well, no, not at all. By the time you see
this video, the team will have added
some operations that do make working with
arrays a lot more useful. See, it turns out,
a lot of developers don't really care
about the exact order that they're storing
things in an array. They just want to use
arrays as a simple way to contain a bunch of flags. For instance, maybe I
have a bunch of keywords that I'm storing for each
of my Charles Dickens books. So what do I do if I want to
find all of my dramatic books, or books that
involve war or crime? Well, we now have an
array-contains querying feature that will search for
documents in a collection where an array contains
a certain value. So I can keep my key words
in an array like this and say, hey, let's
find all dramatic books by looking for books
where keywords, array-contains, drama. So if you're wondering
how this works, you can kind of think of
it like behind the scenes, Cloud Firestore
converts this array into a map, where all
the values of the array are fields in this map. And like a typical map,
every one of these fields gets placed in an index. So asking for books where
keywords, array-contains, drama is kind of like asking for books
where are imaginary keywords drama is equal to
true along those lines Cloud Firestore also added some
new features to add or remove specific values from arrays,
as long as you don't really care about the exact
position of them. So editing this
array of keywords is a lot easier now, and doesn't
run into some of those problems I talked about earlier. So go check out
the documentation if you want some
more info about that. So that's a lot of rules. Can we turn these rules and
restrictions into guidelines? Well, let's see. For starters, put data
in the same document if you're always going
to display it together. You generally want
a happy medium in the size of your documents. Don't make them so big that
you're downloading more data than you need, but
don't make them so small that you're downloading
30 documents from two different levels of
the database just to fill out one
screen in your app. Put data in
collections when you're going to want to search for
individual pieces of that data, or if you want your data
to have room to grow. On the other hand, leave
them as a map field if you're going to want
to search your "parent" object based on that data. Another good time to put
data into a map field is just if you want to put
related data closer together for organizational purposes. For example, take
a typical address. Sure, I can make each of
these fields top-level values in my document, but I think it
looks a lot nicer if they're put into a map like this. I can still query
them just fine, and actually reduce the risk of
keyword conflict naming later on in my document. Vectors and games might also
be another good candidate for maps, something like this. And if you've got items that you
would generally use as flags, go ahead and stick
them in arrays. So that's an awful lot
of rules and guidelines, and maybe talking about these in
the abstract is enough for you to get started. But personally,
I'm always happier if I can talk about these
with some examples that mirror more real-world situations. And it turns out--
minor shock, I know-- Charles Dickens
apps aren't exactly a hot category in the
app store right now. Maybe next year. So follow me along
to the next video, where we'll look at how to put
these guidelines into practice. Come on. It'll be fun. I'm sorry. You're asking for a
$10 million valuation, and the Charles Dickens app
market is already too crowded. I just don't see the numbers. And for that reason, I'm out. [MUSIC PLAYING]
