Awesome Web Data Visualization with Google Maps Platform and Deck.gl - JS Monthly - November 2019

today I'm gonna talk about using the Google Maps developer platform with a new open-source framework called Dec geo which was developed actually by the engineering team at uber so my name is Alex and I'm a developer advocate on the Geo team so Maps basically at Google that's my twitter handle at the bottom so if you have any questions you know about Maps or I actually have a long history and like JavaScript open source all that good stuff so please feel free to ping me if yeah if you have any questions afterwards I realized that my name is very long and hard to spell and pronounce so you can kind of universally find me online on github and everywhere by this angry little dog in the beer box that is my dog her name is Allie she is exactly as pissed off as she looks in that photo before I kind of get into into the actual like implementation details I like to start this talk with kind of a basic conceit and that's that understanding data is really very hard I'm sure so everyone here is a web developer I assume right or an aspiring web developer and which means that you probably work with some sort of data or like data processing or data visualization or analysis in your in your day to day life anyone here that doesn't apply to if you don't get out so that but here's the thing right as we work with increasingly large datasets over time right as human beings we have very little ability actually to really understand and conceptualize what that data represents this is particularly true when we talk about when we talk about geo data so when you look at a whole bunch of different latitude and longitude points for example that represent I don't know like the locations of stores or something like that right the reality is is looking at the data we have absolutely we have absolutely no ability to put that on the map right the conceptualize what that data actually means so understanding data is very geo data is very hard because of this thing that we call the world right so here's an example this is a gigantic array of latitude and longitude points would anybody like to guess what this represents directions path if anyone else guesses anyone think they can parse this the thing is this means absolutely nothing this is like complete guy this is like complete garbage I literally made the made all these data points by like smashing my hands on a number pad like over and over and just to point out the fact that again like we just we have no ability to to look to look at this data it you know either individually as individual points or in aggregate and actually understand what what is represented by it so this is where data visualization comes in okay actually oh man this clicker is like screwing up on me hold on so and up until this point we've basically we've had the pin everyone's familiar with the pen probably and let get me wrong like we love the pin like especially at maps like pins done a lot for us over the years right it's helped us find a lot of things get a lot of directions to do all that all that good stuff but the honest truth is that there are limitations right there's there's conceptual limitations like UX considerations to the pin where once you have a whole bunch of points on the map right like what does it even actually mean to us the the other thing is that who's used the Maps JavaScript API to put a map on a website before cool who has tried to put more than two or three hundred pins on the map well actually quite a few people how'd that go probably like actually not super good right like the the total truth is is that that the implementation of our marker library runs into fairly severe performance constraints once you start getting up to like two or three hundred now granted most people are not going to need to visual to visualize more than two or three hundred points on the map right most of us are maybe doing a couple dozen at the most but again like as we as time goes on and we're working with bigger larger and larger datasets and we look at things in Agra for example right the number of cases where we actually need to render much larger collections of data increases so this is where Dec GL comes in so Dec GL like I said it's an open source framework that was built by the engineering team at uber as you can imagine they probably have a lot of geo data over there that they that they want to visualize within the context of the real world so just a few quick points about it it's a way it's a WebGL powered data visualization framework is everybody from at least familiar with what WebGL is it's okay if you raise your hand and say you're not like I could talk about it a little bit hey so WebGL it's it's an open-source standard it's built on it's built on the OpenGL standard to mimic it as closely as possible and what it allows you to do is to basically access the GPU from from the browser for a rendering for 2d and 3d rendering purposes is the is the common thing so WebGL is what is how Dec GL gets its performance because it's able to push all the render off and the data processing off to the GPU it was primarily built for a Geo or a mapping use case that doesn't mean that that's the only thing that you can use that geo for in fact you'll see that like there's that you could probably imagine a lot of more generic things that you could use it for but that's just it was purpose built specifically for looking at geo data also takes a layer based approach to visualization visualization rendering so what that means is and you'll you'll see later on kind of what like what this actually looks like it's it's actually layered renderings on top of the map and a cool thing is it gives us a whole bunch of really beautiful data visualizations right out right out of the box and I'll show like three or if I have time maybe four of the different ones that just come baked in to deck GL so this is this is really great because it saves you a ton of time in actually making things look nice and there's more so like you said this is a WebGL powered framework so it supports really really large data sets like large remember I said that that we currently run up against a problem with about two or three hundred markers the last time I checked the largest data set that had been rendered on a Google map with deck GL was 6 million so it's a very significant change also comes with a bunch of really nice really nice conveniences right out of the box for us such as the automatic handling of things like animations I'll show you some animation stuff later on and some things that are really important to like UI and and like UI design and user experience that you might not otherwise think about like resizing so for example right if you have markers on a map and somebody zooms way out you probably actually want those markers to resize down right otherwise you can have these gigantic you know icons sitting out covering big parts of your map so there's a lot of that sort of stuff and also it supports the building of custom layers as well like I said I'll I'll show you a bunch of visualizations that that come ready-made in Dec GL but the but the framework also exposes its underlying its underlying WebGL context so you can pretty much do whatever you want with it if you fancy yourself being very being very savvy with WebGL has anybody ever like written any WebGL probably not like too many people right I wouldn't recommend it it's like hard like real hard to like make a box is really hard usually you're gonna choose something you're gonna choose some helper framework like 3GS or something like that like I'm serious if I used to have I used to have a slide in here that showed all the code that you needed to make like a cube in WebGL and it's like a mountain of code so so that the ability the custom layers are something that you can do probably the thing that you would more likely do is deck GL gives you the ability to extend off of to basically subclass any existing visualization layer so that what you and I'm doing is taking something that's already there and kind of customizing it to what you want to do and at i/o this year we we introduced initial support for the for the framework in our Maps JavaScript API so that you could actually use it with with Google Maps now that'll I might have had a beer and discuss and discovered the fireworks transition so so we yeah we we introduced initial support which means that basically like I think about 90% of the features of deck GL are fully supported with with Google Maps currently and it's awesome so this that you see here not the kiddie the thing behind the kitty is a just the basic what's called a scatterplot visualization of all of the street trees in the city of Paris this this data comes from a Paris open city of Paris open data set if you're actually if you're interested in playing around with deck GL or really any sort of geo geo data highly recommend that you that you search for the city of your choice and the term open data because pretty much every major and like sub major city in the world now has an open data program where they put out all kinds of interesting data sets that you can that you can play with so scatter plot layer all it's doing you could probably see is putting a whole bunch of little dots on the map where where there's a tree and then the colors correspond to the type of tree that that's there so the interesting thing about this right is besides just kind of like looking pretty and I mean don't get me wrong I like I like pretty things that's kind of enough but the what's more interesting is that you get like a sense of a place right it starts to tell you a story about all kinds of things right like for example things like how did a city planner think about think about how this was all laid out right it gives you a whole new perspective on on a place so the next question is how does it work right again everyone's a web developer so you'll probably so you'll probably like this answer which is doesn't matter like Carrick's to be honest that's what frameworks are for there the great thing about Dec GL is it extract it abstracts away a whole bunch of complexity in terms of the implementation for us and lets us just kind of get to what we want to do which is to differentiate the design right of of some component of our of our web apps so it's just kind of a it's just kind of awesome but also I'll give a little bit as to how it works so the short answer is this the way that the way that deck GL works is like I said this is a WebGL powered data visualization framework so it so WebGL utilizes the canvas element as the as the way to insert its context is rendering context into the Dom and then it renders into into that canvas element the way that that happens on top of a Google map is in our in our jsapi we have a we have a class called overlay view and overlay view the way that you can imagine it is it essentially puts a transparent layer directly with the z-index directly on top of on top of the map and then syncs that layer with the with the movements of the map because right as you might imagine if you're putting if you're putting an overlay of data of some kind of data visualization on top of the map you probably want it to like move when people pan and like do things when people zoom in and out this was actually a very non-trivial problem to solve later on I'll show a link to a blog post that the Dec GL engineering team wrote about how they were able to work with us to actually solve that problem it gets into all kinds of like interesting but kind of boring stuff around around issues like camera syncing so if you're into that kind of thing can definitely talk about that later on also adopts a reactive programming model for performance purposes and I'll explode it's easier to kind of explain what that means later on when when I show like I give animation demo all right so if we get started this top link it goes to an app that I have running on App Engine and you can actually see all of the demos running live if you are on the Wi-Fi do not go to this link right now because one of the pitfalls of running of running demos for doing 3d visualizations of very large datasets is that it is extremely good at breaking Wi-Fi when like a hundred people do it all at once so I would not recommend if you're on the Wi-Fi going to that right now the second link goes to the source for that demo and you'll see there's like like a big chunk of source code for the like and and demo as well as individual repositories for or individual directories for each layer type that kind of simplify it to make it easier to digest alright the other thing that I always like to point out is please notice that B's links are goo GL e I know that for the last ten years or so Google has has trained us to see goo GL links took about ten years of research and development but we got the e finally welcome to the future and I cannot be held responsible for where Guji l / Dec GL demos takes you I've no idea so yeah feel free to check that out let's actually look at how this thing runs so I have a couple like basic projects here is that big enough for everyone in the back to see you can see this okay awesome all right I'm just gonna run through this real fast so we have a few different we have a few different things that we're importing the first the first is Google Maps overlay so Google Maps overlay is the class inside of Dec GL that actually that actually creates the instance of our overlay view right and then and then injects the injects the canvas element and the and the WebGL context so that everything everything can be rendered next we have a few different layer types so this is this is how the actual like implementation of different visualization layers happens index GL they just they just have these classes and you do and you instantiate with some options and it automatically renders inside of the WebGL context for you and then I'm just importing some map styles surprisingly a lot of people don't know that when you use when you use goo the Google Maps API is in SDKs you can actually you can actually customize the map pretty much however you want like everything from the the the resolution of different labels so like how many different you know place and subway labels this sort of thing that you want as well as the color of everything from like roads to rivers to different topographical features so that you can make really pretty maps and the thing the thing is right if you're doing data visualization or I mean if you're doing web design it all right like you know I think it'd be pretty so that's that's an option the way that that looks is you define you define those customizations in a gigantic ugly JSON blob that looks like this which I do not recommend that you try to write yourself because it's painful instead we have this map style dot with Google comm which is just like a styling wizard so you can see like right I if I don't want any roads to show up on the map or I want like dark mode there's there's like a really fine grain of different different options that are available here for you to do styling cool moving on so we have a basic index.html file there's pretty straightforward for everybody there's a little bit of CSS that I put in line even though I should have put in a separate flow in the separate file there's the the classic div with ID equals map probably anyone who's done maps J's before has seen this right this is the div where we actually like in where we actually insert the iframe into the Dom where we serve up the map to you right I'm just doing an include of the SAP JS file which is where all the deck geo stuff happens back here next I hope I have a variety of data sources these are just URLs to different REST API is for for open data sets that come from Chicago New York City and Los Angeles and then I'm and then in this script I'm just dynamically loading the map loading the map script probably if you've worked with Mouse jsapi before you're used to seeing this script tags like script acing source equals you know api's google.com or whatever but in this case right Dec GL with with Dec GL we need to know that that script has has actually loaded so this this particular function is just is just dynamically inserting that script element and then promise fine the whole thing so that we can be sure that matte that the maps jsapi is actually there before we run anything with duck Jill and then I've just defined a function to initialize the actual map so these are just different center points depending on which which city I'm in as well as different zoom levels because as you're doing obviously right as you're doing different like visualizations or you're doing different data sets into considers probably you want the map to load at different zoom levels that are going to kind of highlight the the view area that that you want the easier to see so yeah so then I just await the loading of the of the API and then this is the basic you know new Google Maps top map that that says go ahead create an instance of the map let it into the Dom show it to the user right make sense pretty straightforward cool so down here this is where the this is where the app actually runs so I'm just doing that map initialization and then I'm creating an instance of Google Maps overlay again remember this is where this is where that overlay view is actually created right and injected and then there's actually a number of different properties that you can get that you can give when you instantiate the the when you instantiate Google Maps overlay the one that we really care about for the purposes of this is just layers so you'll notice layers is actually an array right I'm only putting one thing in the array right now again just to keep things kind of simple and easy to view but and then and then the array takes a set of a layer of different visualization layer instances okay so the cool thing about this though is that because it's an array you can actually composite different different visualizations so this is this is good for a number of things right like you may want to sort of make like put multiple different visualizations overlaid on the same map to create some kind of effect or you might also use this for performance reasons because if you can imagine if I was to load a million data points over HTTP that's probably pretty slow so there's various there's various strategies for how you can handle the performance of this so for example you could come you could chunk your data set and then composite it as multiple layers as an example right so this one is a basic scatter plot layer which is like the city of Paris one that I showed you every layer has an ID I'll get more into that later on it doesn't really matter to this particular instance but it'll matter later on every every layer has to have a unique ID right then I give it whatever its data sources so interesting thing here to notice right is remember that that data source is dot and NYC was just a URL there's actually a whole different variety of data sources that you can feed into deck shell like you can you can pull in a local JSON file which I may do to try not to break the Wi-Fi or a like CSV also the probably the most common is you want to pull in the data from some REST API right so in this case the neat thing is I just give it a URL it recognizes that this is URL and it and it orchestrates the HTTP request and it awaits it for me automatically it's like a small thing but really convenient then we have a whole different a whole set of just styling options for the layer so whether or not there's passes whether or not there's opacity on the points if they're stroked what the fill color is all of this this is a very short list of the possible the possible styling settings you can have for for a layer and all the layers have a bunch so another thing to note is we have like this radius min pixels max pixels that was that dynamic resizing that I was talking about so the these two properties are setting you know as i zoom in what's the big what's the biggest that the points will get and as i zoom out what's the smallest that I'm willing to let them be so you have a lot of fine grain control the next thing to point out is the skin so basically every layer right you're pulling in the data set but obviously Dec teal doesn't know like we're in that JSON blob that's coming back from the from the API to actually look for its lat/long points right that it wants to render so this is where you actually tell it where in the returned object to look so you can see this is a function and it's just it's just iterating every line of an array of data points right and then pulling the pulling you know the coordinates that it wants to rip that you want it to render from there that makes sense and then here like get fill color get line color I'm I'm not dynamically setting the color but also just to point out that a lot of these properties will will accept either a static value or they'll or they'll accept a function so you can dynamically send so like when you saw that Paris one where they were all different trees for example of like you could do that here right you could sit you could just have like a ternary operator or some sort of conditional that says if tree type equals x then color Y right so that creates my instance of Google map our Google Maps overlay and then I just call set map on that and give it the instance of the actual of the actual map div right and so all that's doing is telling Dec GL to this this is the actual that I want to pin this overlay to cool so let's see what that does so I just like have a basic webpack dev server setup here there you go this is like 17,000 data points that loaded that quickly and when you look at it again it performs really well right it goes all the way down to the ground so quite a big difference from the limitation of two two or 300,000 and again I always like to point out that this kind of tells you something about a place so this is street trees in New York City what do you think this big empty area in the middle is it's Times Square have you ever been to Times Square that allowed trees grow in there same thing down here right in lower Manhattan there's Wall Street right not a lot of and not a lot of trees grown around the banks turns out cool so let's look at another one so I am going to save you all the pain of watching the live code boilerplate and I'm going to use the Marilyn the miracle of copy and paste everyone here's a JavaScript of everyone here's a developer so don't lie no you copy and paste I did it too so this this next one is called an arc layer and what an arc layer is made for is to is to visualize the connection between an origin and a destination point as a three dimensional arc all right so you can imagine like you've probably seen visualizations before of like an airport right and all of the all of the flights leaving the airport to their destinations then this is the sort of thing that arc layers is made for again has its unique ID I give it the data source in this case it needs two different two different setters to tell it where to find both the origin point and the destination point some basic some basic styling like the width of the arc and that's it oh wait sorry let me change one more thing because this one is actually in Chicago cool and this is what we get from this one so this is like I don't know like 30,000 arcs rendered which looks kind of cool but it's a little bit hard to actually tell what's going on here right because it's a lot of different data points or a lot of different arcs all overlaid on top of each other right the reason why I like to show this one though is is that this is kind of neat and kind of impressive because like that's a lot of data but what's really neat is when you do this so yeah this is a yeah this is 30,000 arcs that are individually rendered in three-dimensional space and totally performant totally performs well and again you can go all the way down to the ground we're just gonna need I've seen people do really interesting composite effects with this as well like I was at a hackathon and a guy he he took all of this on time and late departure data for flights and he had on time flights as white and late flights as red lines and then when they were all overlaid on top of each other you could tell the the relative on time rate of of an airport based on how pink or red it was so these are the sorts of things that you can imagine doing with like composite effects right nice cool we have just a little bit more time so oh this this data is actually taxi rides in the city of Chicago it's a big like obviously the city of Chicago Chicago is not going to like show people going door-to-door so it's like centroids of neighborhoods so it's like all of the people going from O'Hare to neighborhood X if it was actually going door-to-door then like you have lines all over the place would be crazy looking cool so the next one that I want to show is is what is called a trip Slayer so when we when we talk about geo data right often it's not just like static it's not just like static in time right we want to look at data that's moving through the world over some some time period right so troops layer is purpose-built for this so if you can imagine wanting to for example look at I don't know you have a fleet of delivery trucks right and you want to see how they how they all moved relative to one another at the same time right so so trips layer does and does a time-based animation of geo data all right this this one is just a little bit different same sort of thing though I'm importing my Google Maps overlay actually I don't need this one I keep forgetting to take this this line out I have the trips overlay and my map styling so the way that that I've set this particular this particularly demo up is I use our places API to just do a fetch of restaurants around a center point on the map okay so that returns me ten different places right with their location data and then I use our directions API to to just generate routes that join each each point to the other nine so that gives me like a hundred and ninety different sets of sets of directions okay so this is yeah so this function here is just where I'm generating that place data out of out of the places API and then here in these two functions I'm just generating the directions and then building those all into a race okay same thing as before I load I load the Maps API I initialize my map and then real quick I'm going to come down here so the so the API loads initialize the map I get my places right I generate my my trips okay and then I I create an instance of Google Maps overlay so in this case you'll see that unlike last time right I'm just create I'm just instantiating the overlay with nothing inside of it so it's it's empty right now right it's just a blank transparent overlay with with the WebGL context inside of it and again calling set map the difference is I'm going to call this render function so up here I have this render function and on the overlay I call set prompts so set props is how you can change a deck GL overlay like recursively so for example you know right you don't often want to or there's many times where you're not going to want to just do the one visualization and then just have it do nothing else right you want it usually you're trying to show some sort of change right or some sort of variation so this is what set props is for so you can see that what I'm doing is passing passing it a layer right which is an instance of this trips layer again there's an ID there's data right and then a path the data set for for a trips layer what it looks like is so imagine you have a set of directions right you have the turn-by-turn directions so each turn right is that a latitude and longitude point and there's a timestamp associated with it makes sense so what what what what Dec GL does with in this particular case is it takes this array of all the turn by turns with with their time stamps and it animates the animates the movement between points for us it's like very good that's a very good question so the question is does that mean that that I create a new trips layer on every render so I'll get to this in just a second actually again right we have various settings for what it looks like and then you can see that I have this current time property in trips layer and I'm every time I render I'm incrementing current time and then I'm recursively calling render with requestanimationframe so this is the interesting thing about the about the reactive programming model that's adopted by by Dec GL is that remember I had said the ID has to be unique so what so every time you you give Dec GL an instance of a layer it will use that ID to identify whether or not it has already rendered this layer once and then if it has it will only do a shallow diff on the on the previous version of the layer versus the new version that you're giving it this means that for example you know if it's already rendered out the data it knows the data set hasn't changed so it only has to change maybe some sort of particular visualization property write some sort of like styling property so in this case right the only thing that's changing over time is this curt is the time stamp on this current time which tells it where and time to animate to so real quick I just have I have a little node server running in the background that just orchestrates all of that like places and directions stuff for me to make it a little bit quicker [Music] how does yeah awesome so it's just like give it a second for it to like spin up all the data and that's what we yeah so this again right this is 190 like trips in motion all relative to one another and the cool thing is this is running at 60 frames per second not only does does so I'm only incrementing current time by like one second at a time right but for every frame that I ask deck GL to render it actually also renders three to four in-betweens for me as well so I get these like really nice smooth animations it's pretty cool I've done this I've done this with like 28,000 data points or something like that before another like neat little thing I like to point out is like let's change the color and make the length of the trails a little bit different and render it again and you can see like like your perception of the visualization like really changes based on you know your UI like based on your styling decisions that's pretty cool because right all the way down to the ground if you if you actually have the link with the demo that I had posted before if you're on the trips layer I have I have it where it just continues to grow basically forever so you end up with a lot of different lines moving around the map again these are the links if you want to see it and read it for yourself that all of these demos are made so you can run them on localhost you just run NPM start basically we have some we have some really good docs for the Maps platform also this link goes directly to the part of the deck GL Doc's that talk about Google Maps over way and this is that blog post where that I told you about earlier where they discuss how they actually made this implementation happen oh we just started need a new youtube channel too so if you understand maps check it out we were posting a bunch of videos and stuff there we would love to hear what it is that that you'd like to see as well that's it again that's my Twitter handle thank you very much [Applause]

Channel: Pusher

Views: 2,893

Rating: 5 out of 5

Keywords: developer language, language code, libraries, help, tips, forum, meet-up, meet up, learn code, coding education, coding hints and tips, lecture, coding lecture, learn about code, learn a developer, MAPS, GOOGLE, WEB GL, JS API, DECK.GL, 3D VISUALISATION, ADVANCED DATA, PLOTTING

Id: -wni8jbNUzw

Length: 36min 41sec (2201 seconds)

Published: Sun Nov 24 2019