[Music] give it anything everyone so Peter asked who who for who it was the first time they are at this Meetup and that's me I mean so good good good good so about half of you don't have any quality intergate indicator and that's great because peter also said that he has great two great speakers which means that there's apparently two more people after meeting i thought there was only one so this is going to be interesting let's see let's get rid of some of this junk I'm Frank and I work at Google in fact I work right downstairs here on the third floor in that corner in case you get lost I can walk you out after the event I work on firebase I would normally be wearing a firebase shirt but people swore that I should wear this one I expect lots of tweets of the shirt fine and I don't normally do a lot of talks about firebase and life coding stuff with that but decided to spice it up a bit differently for your for your lots two nights oh by the way I'm Dutch which you can hear from my accent and I live in America which you can see from the fact that I start my talk while chewing gum oh let's get rid of that yeah stay out of the lights I had warned them that I move around a lot so this is gonna be fun hey we're gonna be Wow okay this is the title that I ended up with I said this was going to be something with geo queries that's pretty much what this should cover so let's make sure we do that who knows what geo queries are good if I run into trouble I'm gonna actually ask you for help just so you know I'm not an expert but I'm like a educated layman I would say who is she was firebase before cool I like that I really like that um just because there's so many I didn't expect that how many actually used firebase before we were part of Google cool I think II not spend too much time otherwise I'm gonna eat into aged time which is never good let's see location-based apps I think we talked a bit about that in the in the abstract for this talk in the description who uses lyft or uber wow there's hens not going up but is there like other other ride-hailing services that I don't know could be okay one of the things I like about overrides is that if I open the app it shows me the cars that are around me and then right I tell it to book me an app and it turns out that they're not around me they're further away and then I see that the car is heading in the wrong direction and I can cancel the right all right this is sort of fun that you can in real time see where your driver is so I really love that about it the one on the right is a Pokemon go app now I'm pretty sure this is a screenshot from a while ago because for one there's not that many pokemons and I'm pretty sure we just had an update but also because it shows Britney Spears at the bottom I'm not sure if she's still performing in Vegas actually okay so what both of these heavens have in common is that they show information from a database on a map right and they probably are only loading the information that actually fits in the viewport which is an interesting type of database query on any database but especially on no sequel databases like the ones from firebase um who knows what this is actually it's gonna be a bit hard to see but so these are all the e-bikes for East scooters so this is probably also old because I'm pretty sure they've been banned for a few weeks now but ignoring that this is all the bison scooters in San Francisco I live in Sunset no scooters around my neighborhood unfortunately but so this also right it's just a big database of points of interest and you query them for location I was like wait I know this from firebase because we had a demo for a long time that showed you in real time where public transport was in major cities so this is in San Francisco and apparently when I took this video there were lots of line for teens on Market Street there's also some weird rounding errors because they're not really on Market Street I'm not sure how that works but you can see that these are moving along and this was a demo that runs on the firebase real time database and we query all the objects in this circle and we get real-time updates which is one of the things if you've ever used our database that's what we're known for we shut this demo down a while ago because essentially there's pet builders that are way better it is and we are but I'm quite sure quite a few of them still run on our technology I almost like huh this is interesting I want to learn more about how this works especially with our newest database like cloud firestorm so I was just like hacking on it I was like I need a use case like I was like I wanna write I I imagine I have a map with lots of points of interest on it lots of items and I want to get the items that are within a range of 25 kilometers of Sacramento and Sacramento it's because it's the state capital and I happen to know the location of that that helped me a bit and 25 kilometers because not because I'm Dutch but actually because most of the world works in kilometers so okay so I want to get everything in this green circle and believe it or not but that's what we're going to be spending the next 30 minutes working on myself right I didn't really tell you what we were gonna do so this is the program for tonight please don't take too many notes we're gonna go through them one by one in fact we've already covered number one right location-based apps we know what those are so apparently we're not gonna look at storing locations in the database who has used the firebase real time database okay let's use a cloud fire store okay we're gonna be talking about cloud fire store so firebase has two databases - - no sequel databases our classic one that we started with six years ago I think is the real time database it's unique feature is that it does real time synchronization between clients and we sort of liked it but then we joined Google and Google said we really liked that but we wanted to scale even better so that's what fire store became so fire story is like if a cloud datastore and firebase root end database got married and had a kid that's fire store and this is a screenshot of fire story in the firebase console and to do this type of query enough data on a map right from a database and put it on a map I need a database and it needs some sample data let's quickly have a look at what we have okay Wow we have tiny text if at any point my slides or or my text is not readable just like raise a hand or yell at me because I sometimes forget but I'll try to remember is this readable for the folks in the back no ouch let's see it's the best I can do it stop zooming in so thank you okay so this is the firebase console if I this console was where you manage your firebase projects and I have quite a few firebase projects and this one is called large fire store so I guess it's the project where I store a lot of data in cloud fire store that's pretty a certain thing so you see all kinds of fibers projects on the left so we have authentication database storage all kinds of back-end services to build products we also have crash reporting products like crash live excel we have performance monitoring and we have analytics products and approach that help you grow your app we're not going to look at those don't worry this is not a firebase overview we're going to focus on only one or two of these so one of the databases we have is cloud fire store we also have the real time database but I'm not using that for the demo right now okay so this is the actual day time my database so cloud fire store is a documents database so it stores documents in collections and then you can nest collections under the document again and then you can build a nice tree or for kind of data so on the left you see my top-level collection so I have something called 10 K 1 K and this number I'm quite sure will have a raffle prize for anyone who realizes what type of number this is and then this is the list of documents have and these are the documents that I generated so I generated 10,000 documents and gave them all kinds of random properties really it doesn't really matter all that much what I did so I gave them a color so that I could fill their own color I gave them a counter which is literally I think how often I click on it but it sounds unlikely that I took 6,000 clicks then again I'm a cookie clicker addict so who knows it has an index and a location and the location is the one that I want to look at so what I did is I have a list of all the state capitals and their locations then and actually of all the states with their capitals and their locations and I entered that in to firestorm right how do you store location information in the database if you use the real time database you store them as to to float numbers that's literally it there is no specific type cloud fire story has a Geo point type which not surprisingly stores a latitude and a longitude now I've been having the hardest trying time to remember which one is up and down and which one is left and right who knows leather tooth is up and down as far as I know yeah so I think I got that right if I make a mistake don't call me out on it it's sort of embarrassing so I gave all of these a random location around the state capital data place the document in so I took the point of the state capital and then went a bit up or up and down left and right some sort of box around the state state capital this becomes important later so I have a map with ten thousand documents in the United States somewhere so let's go back to the flights so let's see I think that coffer storing locations in the database I mean not all of them are this simple but this is pretty much the easy one which means that we're gonna look at how we are actually gonna query this and first to do that so how many people have used no sequel databases before you raise your hand cool how many people are still more comfortable with relational databases yeah I used to raise my hand on that it's not the case anymore I grew out of them but sequel is not my friends anymore it turns out but for a long time it was so no worries so the query model on our sequel databases is somewhat different and let's have a look actually at what that means and to have a look at that let's look at a simple database model for a blogging platform right we have a few of those at Google so why not do a use case about it so I have a table of users and a table of logs it's fairly simple right each user can have access to 0 or more blocks and each block can have 0 more people that contribute to it that's what we're trying to do so in a relational model right we would actually create a many-to-many table for that for that relation and then we connect them together so this says that user 1 has access to block 1 so Peter can manage the sfhtml5 block sounds like a good idea 8 can manage the firebase and the enterprise block and apparently I can only edit the firebase blog which feels sort of like I feel left out but okay I get it that one is important to me so this is pretty simple database modeling right we didn't go too much into into normal forms yet Boyce Codd would be proud of us though so that's pretty good so but I need a use case right this is this is just storing data I need to show data so how about we try to show a profile page for a user and we show their name and we show the blocks that they have access to it sounds like something very simple we've probably written this type of code hundreds of times right it's like we start with a pride it's his profile page then we find out that he is user two so we joined the user blocks table write an inner join get all the tools in there then we find all the blog IDs then we join inner join in the blocks table and we find the titles and then we display that it's a lot of words but we've done this hundreds of times and it's comfortable with this right and done this know how its skills also know when I'm gonna hit write reach some scalability problems but overall it's known evil um and that's not how it works in the no sequel database it's very different so my first no sequel database experience is App Engine who ever used App Engine and especially its database appengine was launched ten years ago and this was right back then I thought it was fun to try all this new technology which by the way I still do and I was like sure I'm gonna actually build this blogging platform on App Engine I'm not kidding this was actually something I did it's like okay so I'm gonna store the user who store the blogs right that's great and then I'm going to join them together it's like where do I store my many-to-many it's like then you read the documentation it says yeah you can't do joins and it's like wait these people must be insane right how are you ever gonna get data from two tables if you can't do a join when you need to read it I think everyone who has done relational databases first and then moves to new see no sequel has this experience and it's like okay I honestly didn't know right I was like this is insane and I'm not gonna bother with this platform anymore and then it's like this was before I worked at Google right and it's like wait well these people at Google right there they're not really known to be done so I know that so it's like they must have something in mind right they must know something that I don't know so I started reading the documentation and what they said is no no so what you need to do you need to write every block that somebody has access to in that users records in the users table it's like okay sure then you write every user that has access to a block in that blocks records in the logs table so this it's like okay that's a lot of data duplication this is very not normalized anymore for me right from a relational background it's like oh dear college teachers would cringe at this like but okay sure if that's what they tell me to do right it's like what do I now have to do to show apes profile page but wait I only have to read one record here right there's only one record document thing that I have to actually read like huh I can sort of imagine how they lay this out on the disk and how this is the single read operation all of a sudden oh I think I know how it skills so well right because the the database in appengine was promised to be extremely scalable and it's like yeah I get why all you have to do when I read something is read a single like cluster from a disk I get why that would be fast so writing the data became trickier but reading the data became a lot more scalable we're to use datastore okay so datastore is the database that used to just be part of App Engine so that's now known as datastore and the reason it was brought out of App Engine as a separate product also is because that it's a really good database for scalable read operations which is good trying to figure out what I wanted to explain here this is our document that we want to store so imagine that this is a records in the database right in the database that we're using fire store by the way sort of this spiritual successor to data store I think it takes a lot of the good things that you know from data store and then it adds things like real-time synchronization to it which is pretty awesome right querying that's what we were going to look at so um because I still need to figure out how I can actually query this database so let's say that I want to get all the documents that are in state WI Wisconsin I think Wisconsin cool so all the documents in Wisconsin so this is the query that you would do on the firebase real time database I'm gonna mix query syntax a bit because it's all the same right so we're gonna order all documents by their state value and then we're gonna take only the documents that have stayed equal to Wisconsin which should be if it's ten thousand and have about 50 states two hundred something like that so sure write that that's an easy query to do it gets more or it's got more it's more interesting if I want things like order it just order it so not filtered by the order like I want all these documents ordered by index it's like okay or their order by child index or on and firestore I think it's order by index or something but same thing right you can easily see how how the database actually checks its indexes right finds the one by in this case index and then returns the items in that order but it gets more interesting if we actually want to get the data by state were just for Wisconsin but then ordered by index because that's on a no sequel database is not a trivial operation in fact on the firebase real time database that's not possible and now I wasn't the other side because remember when I when I talked about Google and I said these Dumbo's can't believe that I actually can use App Engine to query and build something and now I have to actually explain the firebase side of this where it's like yes these Dumbo's actually think you can build a real app on database where you can only order by an filter on one property because you can so remember to use case we want to get on the documents that are in Wisconsin or the PI index it's a random use case what if I take the value of the state property and the value of the index property and I glue them after each other so I get a list like this so I create a new property known as a synthetic or synthesized property State index and I just concatenate the values of the states and the index next to it so now if I just put all those like synthetic properties in a list I get state index California 10:47 California a 90 100 all right and then I get Wisconsin 20:41 and Wisconsin 30 86 which I think was a cuter document we were looking for earlier all right i order this synthetic property and the things are then naturally ordered first by state and then in sequential order of index which is like huh that's an interesting one because now if I query this and I order by this state index property and I say start at WI underscore which is the first Wisconsin and then and at W I tell them and now till there is not a magic operator till there's just the last principle ASCII character so everything in this index is before that like so now I get a range out of this data set that essentially includes everything in Wisconsin in order of Index sure I had to pretty much write my own own database here but I think it's pretty cool and this is what we've been telling people for years to do this is how you actually can filter documents on multiple values but still a single property so you're essentially cheating the system and that's sort of I think the theme of the entire talk is that we're cheating the system doing the impossible by coming up with cool data structures not the coolest data structure but still firestore actually is a lot better if I story ever want is I just tell it that I want this so in fire store I don't have a location I have a collection I say we're state equals Wisconsin and then give me the results ordered by index which many people think like ooh this is magic fire boys fire fire stores query model is so much better and I definitely agree that the query API is a lot more friendly but the query model is exactly the same because what fire store does behind the scenes it actually generates this synthetic index for me so it generates an index that consists of two values now I've been telling this quite a bit and then thought crippled men those Todd Kerpelman he he did the videos here he's on fire store and he called me out he said that I did it wrong because he says that it actually doesn't generate his index it does something that's known as a zigzag merge join I mean it's a toad calls him Z TMJ because they're friends they know each other well so but apparently it is able to do this from a single index that's pretty cool actually so it's very efficient and the thing to keep in mind with a no sequel databases is that everything is focused on scalability of read operations everything else is secondary and what's important about query query in these databases is that all the results come from a single range so let's see yeah I still have this one in here let's make your back to this one so remember this quite complex sample right it's like this is not fun to build but if you now think of what the database has to do to give your result right it loads this index right this this property for all the documents then it finds the first one with Wi underscore and from that moment on it knows exactly what it needs to return to me right it just starts returning results can stream them to me until it finds the end of the range that I requested so these are known as range queries and they are the reason why we can scale these so well and there are also the reasons why the query model is not what you're used to is it's very limited if you use two relational databases or for that matter many no sequel databases that added capabilities so real time database and cloud fires too were very much will only offer you api's for things that can scale not even linearly not even longer with rhythmically they literally scale by a constant so the marketing verbage for fire story is that the query performance depends on the number of items you request not of the number of items you request them from very difficult but what it means is that if you request 10 items and you know how long it takes good you now know how your app will perform doesn't matter how many documents exist in the collection right when you're testing is probably like in my case 10,000 doesn't matter if it's 10 million it doesn't matter if it's 10 billion if it takes one second to request the 10 documents when I have a thousand it would take one second when there's a billion documents that's the guarantee that firestore makes and that's why it doesn't allow you to do certain types of queries because we can't deliver on that guarantee anymore which is all fun and all lots of product mumbo-jumbo it's great when you're talking to the engineers downstairs but we had a problem to solve because how do I query by location how do I get those documents in the range of 25 kilometers around Sacramento remember that's where we started so this is a location I'm not sure if it's location around Sacramento we'll check it in a bit but it's a location and if we do what I just told you right we can combine these two into a single string you can just glue that together and I can query for it right and can create for both and it's like wait no no I can't create for both because I can get everything that's between 40 and 42 right let's assume the first one is latitude which it normally should be then I can get everything between 40 and 42 but then I can't filter on the second number anymore right there's no way I can now and still filter on longitude like oh but that sort of sucks so let's try something else what if I take everything on latitude for 167 until 138 all right and then I can filter by longitude right then I can do a range filter by longer too it's like yeah that doesn't help me very much either like so it turns out that is it with this constrict you can do many equality operations but chenkin you can only do one range filter that's actually one of the real limitations of both querying on real time database we haven't found a workaround yet and definitely also on fire story you can do one range operation or arrange range operations on one value and not on any others you can do equality on many of them so that worked great when we did Wisconsin and index but doesn't work anymore when we need to filter by location because I can get everything in a certain strip around the world right but I actually want only a tiny circle here so that sort of flight sucks right because we set our cell to use case ensign hmm we're failing so luckily somebody found some unicorns and found a way to actually make them turn out magic and the magic is known as geohashes who has ever used geohashes anyone who said yes to Jo querying must by my definition raise their hand oh cool kojirou hashes to me are magic I started investigating him as like these things are cool I mean I like to geek out so let's have a look at what the geohashes because remember right we're still trying to solve the very simple use case of documents around Sacramento so we need to find a way to query two numbers where actually the database only supports querying one thing so we need to somehow mesh two numbers together into one thing and that's what we're gonna do so let's imagine that we split two earth into two pieces let's say one in the top health the other in the bottom half let's for sake of argument I like difficult names called that line that we defined that we draw in there the equator or something right just a random line that we've drawn on the earth now we're going to say that everything above the equator is a zero and everything under it is one it's like cool now with one bit I can identify half of the earth right so I can say that my document is above the equator or blow it okay it's nice now let's see let's do the same for east and west so left and right so that's the Meridian I had to think of what it's called so that goes through Greenidge and let's say that everything left of the meridian is a zero and if everything to the right of it is a one now we get this now I have a two bit value that allows me to identify in what quadrant of the earth a document is that's like pretty cool but especially it's pretty cool if I then realize that I can do this for every quadrant again right so now I have four bits and it allows me to determine in which of sixteen areas a document is located it's like wait that's four bits that's one one nibble it's like that that's actually pretty pretty fancy especially since I can find the region I can keep repeating this so a geocache has all the properties that you think of from a number except that it's not a number and it combines two numbers in the same way that's a look at how that works so this is like sort of the idea behind behind your hash is it actually is a lot more complex because they don't use just bits they use an alphabet of 32 so they have a 32 character alphabet and this is the first level of the geohashes on the earth with actual UI values so if you look carefully you can see that at the bottom it has this little quote that we had earlier 0 1 2 3 remember it's similar I think I might have inverted them but ignoring that then we move to the right right and it has 4 5 6 7 then we go to the top left all of sudden it's 8 9 BC so we're missing 0 and a so GUI should use a dictionary that consists of numbers and lowercase letters and then they remove the ones that we're probably going to be most confusing and 0 sounds like a very good one to remove the a I still don't really know what they thought it would be confusing with but what is that there is a q0 so wait oh then I know which one isn't there then they're always in there ok that would explain ok I still don't know why they removed to a by the way yeah okay got him ok so we're gonna keep going for this one you see that we don't have the oh we don't have the I because those are definitely going to be confusing and it goes all the way up to Z at the top right so with one character we can now identify 32 regions on the earth I'm not sure if you can see definitely you can't see it in the back but this these tiny boxes over this this is actually a tool I will see it later where you just click points on the map and then it shows you the geocache for that and since I don't just click on the entire n I click on a tiny box in there you can see that at the bottom it says n1 is where I clicked so that's so sub-region one of the N box and in the you the thing that you have there is us which is somewhat confusing this that's not where the US is but if we ignore that semantic mapping then that tiny box is the subdivision s of the region you this is pretty cool because we can keep doing this right we can keep dividing the earth into smaller and smaller regions that are identified by longer and longer strings and that's actually fun because now if each region is a single string and the longer the string the more the smaller the region becomes the more detailed my knowledge is I can actually query it with firebase this is a list of the size of the region the cell as it's called based on the geo edge length I got this from from Wikipedia but apparently our one character geocache could identify regions of about 5,000 kilometers which is like oh it's a nice first start right but when we get to eight digits it's 38 meters by 19 meters so I'm not sure does anyone know how big this room is don't worry we're gonna find it out soon but once I get to 11 or 12 digits I'm sorry I it didn't fit on here anymore it's like wow this is some precision work that we're doing here and keep my that's the string that okay I can't really remember it but twelve characters so it wasn't digits twelve characters is actually not that much data so geohashes are a system that allows us to take a location right which consists of two numbers and turn it to to a string that actually then has two but both those numbers in one value and that value is magical because from the left of the value as you move to the right the meaning the significance gets lower and lower which is something that we really like about numbers and it happens to also be the case for geohashes it's slightly harder for us to reason off because there are strings and not numbers that's pretty much the only disadvantage of them okay who knows what this is really this is here I mean I didn't know what room we were in so I took a broad stroke so this is where we are so it's a bit hard to read but it says 9 q 8 YY z w - that's here just so you know this is where we now live okay let's actually have a look at that let's look at some of these properties about geohashes okay different tool that shows geohashes wait I need to actually make sure that I copy that value also wow that's just jump out of this for a moment I have no idea what it became hard we're opening the tool again so not sure where it starts probably where the author is just like we start where we are located tonight this might look slightly more familiar especially if you've flown a drone over our building not sure if that legal by the way but not sure how we took this picture so don't know okay this is pretty cool actually now this is sort of almost half of a floor and this room is big but it's not half of a4 yet so let's see if we can find us right so let's add something to it let's say that we add a 1 okay now we're it down on the balcony but we're actually not on the balcony because there's no balconies on this floor so we're floating in the air so let's be careful with that let's not do that so let's do it hey wasn't part of the alphabet B okay so you can see that all these boxes are still within the box that I had earlier but they're smaller right they're a subdivision of this so C is probably right next to us right and it just goes through that so I can find tiny boxes right this is a small region and we keep kicking the I don't know how many characters they support to support 9 but you can keep making smaller boxes if you want right we can find where the the bar the mixologist is actually keeping a secret ingredients if we want okay so let's zoom out a bit so okay this is where we started cool now we pretty much have all this south part of amber Kadare oh that's pretty cool we're almost at the Ferry Building okay so this is south of San Francisco or the southern part of San Francisco sorry apparently doing San Francisco and half of this is the ocean is about four characters it's a geocache of four characters right and we keep going and in the end when we have a single character this is pretty much what we started with right remember who remembers where zero was yes zero was very cold apparently even the Penguins get their own geohashes it's pretty awesome okay but this how do your hashes work does that make any sense good because if not I would have to send you to somebody who actually knows this stuff cool let's go back to the slides okay so we need to store the geocache into a document now if you paid attention earlier I actually already did that but let's see how I did that right so how do you encode a latitudes and longitudes into a geocache value ah we even have a topic for that it's probably the shortest topic in here so how do we take the value from the left and encode them into where we are so this is apparently delighted to the longitude of where we are now and it's like wow it's this piece of code this one I can zoom in I apologize but this is all that it takes so if you would refute op-232 character dictionaries in here it has the maximum ranges for latitude and longitude in there and it then goes through a loop because you asked a certain or certain precision in this case apparently we encode them as ten characters did anyone pay attention and see what the precision on that place it's probably a few meters is my guess something like that so that's how we encode it so we loop over the latitude and longitude that you give us I'm sorry we continue looping until we have the length that you asked for and then we alternate latitudes and longitudes so you see an odd and even check there right and we take location 1 longitude or location 0 latitudes we just take the next bit of that and we encode it into our geocache and then when we've reached length we return to geocache this code by the way it comes from a library called geo fire which is the library that moved that we use to show those moving busses on the on the map of San Francisco or major metro metropolitan areas around the world and I just copied it from there like nothing that I did here I didn't invent you ashes I didn't write this code I'm so lazy ok so this is what we need to store I think you have a look at some of our documents sorry no let's have actually if I look at what this is another site now this one doesn't show the precision of what we got but this seems to be where we are right so that's actually pretty good so our geocache translates to to dislike the tubes and longitude which I think is correct you can also see that there's fewer numbers on the latitude and longitude so it's not very detailed but it's hard to see on this side let's keep going because it's more fun I think we just did this let's have a look at geo queries now this is where we get to the really tricky stuff it's actually I made this as visual as I could because this was fun and the reason I'm actually giving this talk is because we were looking for topics here and some of my teammates said well you've been like talking our ears off about geo queries and geohashes so you might as well do we'll talk about this not a bad idea so okay geo queries just in case you forgot we know we've been here for a while we want to find the documents within a range of 25 kilometers around Sacramento what we started with we now know how to associate a strain with each document these are the geohashes around Sacramento so you can see that's it's 9q b c and f because DNA are above it and then you can see that we have all these four character geohashes in the middle of that because everything that we need is currently in GQ c which is actually not hard to remember so apparently a three character geohashes something we can remember in fact if you look carefully where a Sacramento I think it's somewhere right of the seven can you see it's like it's somewhere around there like okay so in fact we need this box pretty much right this is a good starting point of what we need now remember how we query we need a single range of values from a single property so that's cool because if I now start at 9 QC 4 that's the bottom left now go all the way up to 9 QC V then I have everything that I need it's like whoa that's interesting that's like a type of query that I can do on firebase it's like I think we're done which it's not really true does anyone see what's wrong with this query starting at 9 QC v and ending at 9 9 QC for sorry an ending at 9 QC v exactly good points so you can see that a key goes outside of the box because remember we have like just a range and we pretty much randomly encoded them not really random but there were these quads and I love the quads until they started doing this to me because now they're in my way right it's like now apparently the quartz from 9q CH and stuff goes up there you can see J m and then n all of a sudden it's like that's in the way don't like this so it turns out we cannot do a single query for this which sort of sucks but let's see what we can do and in fact it so this was really my first test was like my first test was wonderful because I picked Sacramento and it seems that Sacramento is pretty much at the edge of four of these quads it's like it's wonderful it's like everything that you can pick wrong about it was in my first use case I mean luckily it meant that once I solved that one everything else seemed to work fine so what we're gonna do is we're gonna break what we need to range of what we need into four we're gonna actually query each quadrant separately separately so we're gonna take everything north west of Sacramento everything north east and then also to the south we're gonna just do four queries instead of one turns out that sort of takes some math but luckily chia Fire has a has a function for that so this actually determines the bounding box if you give it a center and a radius it will give you the four bounding boxes for the quadrants that you need to query so and as you can see it literally just determines some degree math and then converts it to latitude and longitude again and it gives me four ranges essentially four yeah they're literally ranges I cannot make anything else of it it's like let's a kick yeah to this one so it's the left of this box to the top right of that box is one range then we get this box the top right of that box right so each query has a start and an end so it has two latitude longitude soar to geohashes here it was still latitude longitude um these are the ones actually so let's have a look at them on a map and I color-coded them I probably tweeted this one and people thought that I was playing games but you can see that I drew them by hands because they're not very straight but these are actually for queries that we do and I try to indicate which one goes where the yellow is a bit hard to read but it's 9qc d29 q CH is this quadrant ok interesting we'll see something there 9 q CH 9 q CN is here and then we have the other two colors also it's two interesting things about this I will take answers by the way who sees some some interesting facts about these queries no worries I will give them one of them is that there's these two weird ones here at the top and I don't really know why they show up but they consistently do many of my queries have a slides outside of the box thing somewhere I can't really figure out why so chose to ignore that because it's more fun to worry about interesting things the other thing is that they overlap so remember the first one goes to 9 q CH but somehow the red one also starts at 9 q CH because the quadrants right remember it's actually it seems like this is 1 2 3 4 in the order that we had initially liked so these overlap and that means that I could actually merge those two queries together now I haven't done that but geo fire does that for example and then you only have 3 queries that you have to do and you have fewer duplicates by the way because that is the issue with this I get surprisingly few duplicates by the way but not sure why okay I think we're almost ready to start querying remember it all started that we wanted like this tiny circle in the documents that fit in it this is actually sort of like the 25 kilometer around it so we still have one problem to solve because the boxes for outside of the circle is like so I'm gonna have documents that are in my geo queries but that are not in the range that I want to return who knows what a haver sign is good I didn't know either so I thought it was true haver sign I thought he was a famous English like mathematician from like the Middle Ages it turns out that the haver sign is a type of sign like it's actually a ver sign or something it's a very odd mathematical thing the important thing is that the haver sign formula which is this one and you know where it comes from because it comes from geo fire actually allows you to determine the distance between two locations on a sphere that's pretty cool because that's literally what we have right we have a sphere it's the earth right we have two locations the thing that we want the center of the query and the position of the document and we just need to know whether it's less than 25 kilometers so we're gonna pass all the things that we get out of the database we're gonna get there actually letter to the longitude pass it through this formula and then we know whether they're in range and that means that we finally can actually get the documents within 25 kilometers of Sacramento which is this I can't say I was impressed when I got it it's like okay look I got markers on a map but this is literally the documents that we were looking for since I know that doesn't get a plus and it talks like this we'll look at it life don't worry but I quickly want to explain what we're seeing here so the green markers are the documents that are within the range that we want so these are documents that are within 25 kilometers of Sacramento the red dots are returned by our geo query but are not within the range that we want did anyone quickly count them by the way because I of course did because why would I write code for something I can do manually it's like so it's about 25 green markers and about 50 red markers so we three times over query I wasn't too impressed with that but still it's better than taking the entire strip of the earth and then only getting the circle so I think I still won quite a bit there um looks like you look at this life it's way more fun now I know that you can't read the code in the back I'm gonna zoom the code for a brief moment and then we'll get back whoever used J's bin or jsfiddle for that matter cool I use them a lot because I like sharing code with everyone I also like not having to install anything it's quite amazing who uses stack blitz by the way or glitch or like a cool and hard time getting this in stack stack place otherwise I would have done that so I pretty much jumped all the code that I need into a J's bin and then made it work it took a while we had people used firebase if you use firebase on web then you know that this is initialization bits you also know that I'm not afraid of anyone taking pictures of this screen that contains clearly secret values so these are not secret this is actually how you configure your codes to find your firebase project on our servers like there's nothing secret about what I'm showing you now I think maybe the fact that I should really support you as been that's fine okay so I initialize firebase now where it's oh and I have to generate data generator which is also a J's bin I generated the data from there now don't worry I closed off the database after I generated the data so you can't be adding data to it now but even if you could my queries would perform at the same rate as we had before one of the wonderful things about fire story ok gos you tale so this is a very long file we look at some interesting bits for a moment but let's just skip over it for a moment so gia has utils is the main worker file that I used from geo fire and you'll find an open source library so you can just find it and if you have any questions about it I'll be sure to find someone who can answer them it has all kinds of things like converting latitude longitude into geo hash finding the displayed of the box and simple things like caverna converting letter to the longer to two meters like bounding box bits bounding box coordinates this one we just saw on a slide ok I'm looking for the end of gos utils there we are some stuff that we don't need then there's the remember survey for sign this is for his formula and so that one came from a different file apparently this is the code I wrote so I know it wasn't a lot most of it I copied from some we're but still this is code that I really wrote it's like if you've never used fire story this is how you access it you get fire story from from the firebase object and then you ask for a collection and remember my collection was told what's called 10k because I have 10k documents that I spread out over us then I have a thing called States let's see we don't see use copper if any idea like this is like Alabama then named Alabama abbreviation and I have population have a capital guess what I have locations of the capital this line is really really long because I have 50 states and all this information freedom there believe it or not this took me the longest time to get because I couldn't find a good Jason for it so I created that one ignoring that I have some HTML stuff let's ignore that not all that important in the end sorry I have two boxes here I have a big estate which is empty at the moment because I didn't run the code yet and a half distance if I change either of these I'm gonna draw maybe I should just show you run with Jas I said if I pick a state it's gonna draw the items around it not this is not 25 kilometers which is what we promised so that's the one but this is what we started with now this is where I will hold for applause [Applause] okay whenever you change either of these we do show map for state by a state and a max distance yes I also have no idea why I left that uppercase swath again so I do something called good weights get queries for documents around so I have a state and I find queries for the document around something the center is actually the states capitals location so that's sort of I hope makes sense so that's a function I wrote which takes the ref the center and a stadium and the referee is the collection by the way that we are querying so it's just some fire fire store reference so then I do I determine the geohashes to query which I think I should have somewhere around here actually don't know sorry I determined them from gos queries that's what I was looking for so that's a function where I passed the latitude and longitude and the radius in kilometers all of a sudden or no in meters sorry but ignore that and then we get back the ranges that we need to query so remember we had the four colored boxes wow there they are okay so this is gos query center and radius we do bounding box which we boot bounding box coordinates which you might remember it's how we split things in four pieces and then we do gos query returns the latitude and longitude into a geocache like okay let's go back to the bottom where we actually run this code so I get a number of firestore queries so sorry I should show you five store queries so I loop over all my geohashes at any two query and remember these are ranges from the bottom left to top right I think it might also be the other way around so we do a ref tot where to your hash greater than location dot zero remember the geocache contains the latitude and the longitude so we've just done some magic I hope all the unicorns are still alive but we got the magic we wanted out of them so I'm not complaining so we do the geocache greater is there greater or equal then the first one and then smaller than the second one and that gives me a query to fire store that filters on a single field that - values I do all of those and then I get queries we loop off the query we do query would get so this actually execute the query on on the firestore back-end now remember I have 10,000 in there it I'm not sure why I actually named my collection by the number of items in there because remember it doesn't matter how number how many items are in there I probably had the idea that I was going to create all these collections with more and more to prove that but clearly not what I ended up doing so but this gets me however many results there are and that's the performance I'll always get so we have a Google map we loop over the query snapshot which is all of the documents that resulted from this and then we do a distance calculation the heifer sign formula and if it's within the distance then we do some calculation not all that important but we add a map to the marker so I add some stats here that's pretty much it this is the entire code that we have here like I admit it wasn't too impressive when I created it either but what is fun is that you can now play with the values so California California who is not from California cool what state okay could also have been somebody not from the states and sorry that's all I had in here what's the capital of Vermont thank you and can't believe I this actually is hard to see we believe that the code is running right I mean that that we're not doing anything fake here so I mean it's gonna pop it out into its own window because just makes it easier to see oh right after we run it so we I can't really make this a lot bigger saying kilometers is well this I can definitely make bigger this is not good so let's do that 25 kilometers around one per yeah you said nice okay so these are the documents that my remember my very very intelligent algorithm of placing documents on a map which is called math - Brendan replace them around the capital of Vermont Montpelier how many people can pronounce that properly the advantages of being European so that's pretty cool I'm not sure why actually there's nothing here in the top right I hadn't checked Vermont all that much yet any other states that we want to check I have a few that I want to check for sure I heard something but I couldn't Marilyn actually that's a good one and I like how none of them are on the list of like states that I definitely want to check because they have some interesting things like okay Maryland's you can see that I'm not all that picky on where I place them because half of my documents are in Washington like and I'm pretty sure that Washington is not in Maryland reason for that by the way is that literally it was random I didn't do any checks on it I literally just place them in a box around it capital of Maryland and guess what this is where they ended up okay 25 kilometers less 250 oh by the way here at the bottom sorry I should have called that out we can see the number of documents that were returned from the query and the number of documents that are within range that ratio is very interesting for me because it's how much we over query so we apparently got 79 results and we wanted only 23 to 23 is Green the 79 is green and red combined is all the markers so it's like okay that's 3x over query and keep an eye out for that number so if we do 50 kilometers for example then I get 544 documents an 81 were within range so that's definitely not three times over carrying anymore that's about six seven times over querying so not good but still better than remember taking the entire 50 kilometer strip and doing that so still pretty good what I found as I was going through the states is that this box in its shape is very different in different places what's the northernmost state Wow good that was not my first guess myself yes Alaska I need know what's different about this it's portrait mode all of a sudden rides so apparently Alaskans like portrait month no does anyone know why because we've seen this difference in shapes before yeah it's further north rides and when we looked at two geohashes initially remember that the boxes edited poles were a lot higher than the boxes around the equator that's what we're seeing here that's a key cool what's the southernmost state I love speaking for Americans because this is where Europeans would have no idea it's like indeed Hawaii let's do that one like how many documents do we think we fit in a why by the way just out of curiosity yeah it should be about a 150th of ten thousands of course so 200 it's like it turns out that I didn't check for water so very few documents are actually on Hawaii it turns out but I mean still these are on Hawaii also see how this box is a different shape again now part of this is to geo querying algorithm part of it is my crappy algorithm of placing these documents because math dot random apparently also just gives me numbers and doesn't take into account that the earth is curved so um last thing I want to show last interesting use case is actually no let's do New York first for a moment so New York and let's do 50 but it was a great like exercise to actually find all the capital state capitals it's like I was like are they sure that's the campus moments like yep turns out they were and they were right we'll have an interesting run for that in a minute so 50 kilometers around opening yeah it's old money here it's like okay so that's pretty cool let's go back to California where we started remember 25 kilometers around Sacramento is where we started this is 50 kilometers around Sacramento yes thank you okay this was my first test just so you know so the map that I showed you initially was not the first map I got this was my first one it's like holy crap what's going on here something is broken right like something is weird I saw two weird things here who can call them out good one so definitely the gap right is the first one it's like why is this book so wide right I noticed that now as we go through the states which I didn't do I spent four hours for every in this out by the way so if if you got it rights sooner than that then definitely job said google.com is a good idea but it's like why do we have a gap in the middle second I honestly can't figure it out and it's like remember my algorithm I place the documents I repeated it intentionally a lot at a random distance but in a box from the State Capitol wait what state is to the right of California that's Nevada so these are the documents around the Nevada State Capitol what's the Nevada State Capitol know yeah I really did not know that by the way so it's an it Carson City that's right here now the documents to the left here look very similar to the documents to the right here right that's the same range of documents that I have here so it turns out that the geo queries working fine it's my data set that's broken because I placed them in a box around Carson City and Sacramento and it turns out that the boxes don't overlap there's a gap between the boxes but since I query for the range of 50 kilometers apparently Carson City is pretty much within 50 no not just not but within 50 kilometers of Sacramento it's like okay I did not realize that so there just is no data in the empty bit in the middle like that's what happened here like wait this is weird right how broken is my data set and that's for me always even more interesting I broke something now let's figure it out yes Network thank you let's go back to the actual jsbin because one of the things I liked about you has been remember is that I can change the code so let's see let's show my HTML there we go so this is my drop-down and the biggest range is 50 kilometers where's the fun if you can do a broader range so let's make an option that has I'll bake art estates and kilometers let's let's start with 5,000 and we'll pick a state in the middle okay let's run this again it's actually height to JavaScript and HTML 4 in a moment run DJ s hmm let's load it again what's finicky don't know why 5,000 kilometers what state is in the middle Kansas I'll I'll take it why are states not showing what did I break did anyone see what I broke no no no jsbin might be American but that that's not though I needed a not the code did I forget to close my option or something was that it what was that I clicked it right I did not okay thank you that was not my best okay 5,000 kilometers the reason I actually missed that one is because I'm incredibly nervous we said Kansas 5,000 kilometers is about 3400 3,200 miles something like that I think yeah so it definitely covers the entire United States I've driven it twice I know by now okay does anyone want to click for me because I'm gonna run away when we do this because we're gonna be loading 10,000 documents so okay takes a bit longer remember it does take longer to get more documents from the database it just doesn't doesn't depend on how many documents we have cool we loaded 10,000 they're all green which sort of makes sense let's zoom out a bit okay wait we're still missing we're still missing something there yes you know what Hawaii is not within 5,000 kilometers of the capital of Kansas like something we learned today so this is my data set and if we now assume in you can easily see that we actually do have a gap between these two lesson to be learned here is that you should always first test with the broadest part of your data set and only then zoom in now actually lesson is that it's fun to explore in heck these things so this is the gap that we were seeing earlier it started with such a simple question remember all we wanted to see was things within 25 kilometers of Sacramento it was it sounded so simple all these apps make it seem so simple in the stories because some people do this magic work behind the scenes I think we just covered all of the interesting education educators I'm not completely sure but I think I had most of them [Music]
