"What in the World?" GeoDjango presented by Tyrel Denison at FayettePy

doesn't have a gun to do that one nice Congrats record started I did it alright cool so Tyrell tonight is going to talk to us about geo jingoes I really would get started nice alright well I'm gonna do all the screenshare shuffling so you got to bear with me I'm also gonna try and clean up my desktop or sweep everything under the rug I need my when we do our dot file sweep yes yeah I need that scripts that cleans the desktop yeah I need that script bathtub done SH there it is it's a little different than the web guy versus cells dude yeah it's the opposite of that think no that's not the one I want on this one so nope yeah sorry share alright okay so can everyone see a nice PowerPoint slide there what in the world what in the world indeed what in the world happened to my mouse that's a better question the Python he did no kidding there it is it's back there I am so hey guys for for any of you who don't know me which is a couple of you I think my name is Terrell Dennison director end of engineering lofty lofty is a software consultancy in Fayetteville again there's some of you may not maybe it may be your first time or maybe you haven't spoken with us I'm there Casey whose isn't a Chad is also from lofty and we're we helped coordinate these meetups with Chris here Casey does so glad to have you here I am part of the the mess that is the podcast that our software consultancy produces both Friday afternoon employ and if you've not heard it you should check it out Friday higher lofty calm it's actually a great resource these days because so many people are full remote that perhaps weren't before and maybe you're missing kind of the ability to surround with other developers and complain about JavaScript or complain about Python or complain about docker or anything like that you can come listen to us complain and be like absolutely I totally agree with that or you know you guys are idiots and you can say that out loud or then leave a comment anywhere or just email us because there's my email you can send your hate mail there but we a lot of the ladder we hate than the agreement so yeah anywhere I'm social I'm at Tyrell Dennison I'm sure you can find me there as well so that's not it there we go Maps what are they everybody loves maps these days part of as data is becoming far more like the key to everything in the minds of a lot of people and realistically it is providing so much insight around so many things to be just super vague and businessí ways to visualize your data are key and maps are a great way to do that everybody loves when you're asked to produce a heat map or things like that and pictures do pictures with it pictures speak up thousand words pictures yeah that's a that's a very true statement like you can visually understand things more significantly than you can with words and so there's something powerful about being able to visualize things and so the the good news is as Python developers and this tool is certainly not just used for for visualizations in fact it's way more powerful than that and we'll only talk about the visualizations there towards the end but most of the time the ask would be like can we get that on a map a client or a stakeholder in a company and feels like that we've got geodjango and I don't know if any of you have experience with Geo Jango yet we have had some minimal experience with it prior to a couple clients ago that really came in and and needed to leverage a lot of geospatial information and the great news is Jango had geo Jango built in in the contribs and so to kind of give you the straight from the docs version of Geo Jango you can read along with me Geo Jango intends to be a world-class Geographic web framework the goal is to make it as easy as possible to build GIS web applications and harness the power of spatial enabled data this is the way they do it you get Jango model fields with OGC geometries and raster data you get extensions to Django's RM for clearing and manipulating spatial data it loosely couples high-level Python interfaces with GIS geometry and raster operations and data manipulations in different formats and it gives you the ability to edit geometry fields from the admin so you're probably going what the heck was all of that and that's understandable because when you enter the world of GIS there's so many acronyms so many there's a there's just a language to it and I mean we're kind of accustomed to that is developers but anytime you enter into a new sphere you're like I've got so many acronyms and words to learn now to be able to speak the language of this this niche market or this this technology and so the same is true with GIS software and things of that nature so I will do my best to debunk those as we go along bear with me along the journey if I skip something or don't immediately tell you what it means I'm pretty sure I answered most of those questions through the course of the talking them obviously we can chat after or not QA I talked about stuff so I mentioned OGC that's the open geospatial consortium basically it's it's the the open source source ish body that determines how we are united around geospatial information kind of like Latin long and and other things that we'll go into like s our IDs and things like that but basically they're the governing body they kind of keep everybody unified around geospatial data next is GIS you may not have heard of that list geographic information systems and that's super broad it kind of can be thought of like for me initially I heard GIS and I heard it tied immediately to post GIS which we'll get into in a minute and it just assumed like that's it that's the term that's the thing that's a really specific term now but it's not it's really broad but it refers to systems that deal with geospatial data i'm geographical information systems so looking at the components of a geo Gengo app there's that post here so I was talking about posters databases opposed to Stata base thankfully for all of us is nothing more than a a Postgres database with the post gist extension installed and that gives you some methods there's some topology stuff that gets added on there that geo jingo doesn't take it advantage of but once you get that extension bolted on to coast kiss up to post Coast excuse me you know how they post just database and everything goes exactly the same with a handful of extra functions that you can run at the post use level thankfully for us we're working in Jango those posters functions are wrapped up in our M functions that we'll go into a little bit but then there's also like a basic app configuration that has to take place so if you'll bear with me we're gonna try and hop over here to some sample ish code you're my editor with me now or I can I get a thumbs up or something sweet okay cool so looking up here this is I mean it's for setting up your database once you've got that extension installed it's as simple as defining your back in there again like I mentioned it's baked in so it's Django contribute me see if I can increase the font size on this real quick if you enhance it is my enhance button it's command + encode that would make the most sense from there enhance and as such chrome adds just Chrome so yeah so right there in the contribu have a GIS which is where all of that stuff's gonna live you get your DB backends post gist and there's actually several different geospatial adapters that post that geo django has but the lion's share of people working in django are doing it in Postgres so that's what I've built my stuff in for this example what we've built our production code in and what most people will kind of be most familiar with and then additionally you'll just need to have it defined in your installed app and add that contribute in stalled apps and that's the basics of what you need to like have the that kind of functionality in place obviously the the basics barely barely get you started there's a tutorial that is super helpful that's on the main Django project website about geo Django and we'll see more kind of examples that that were birthed there and we're implemented in the code I've got here but you can go read there and the warning I'll give you is they talk about it all like it's very simple and and fundamentally it is but we're all at least marginally aware as developers that people can kind of hand wave things that are more complex like once you understand it it's rather simple but on initial builds is far more complex so you'll have stuff run and you'll run into like you might have fun getting that extension installed in Postgres or things like that but we'll go through kind of all the shapes of agio django app through the course of this tutorial so you can kind of see how it shifts a django app and what to do with different things but backing up and looking at kind of the Keith's stuff you want to deal with here is geospatial data if you didn't have geospatial data you wouldn't be using geo Jango anywhere so there's different formats for geospatial data Gino Jason is is increasingly becoming popular a lot of tools use it especial with spa apps that people are building there that are geospatially aware and things like that it's pretty performant and obviously Jason is consuming the world along with JavaScript so that's a pretty regular format geo packages is another one that that can be ingested geo packages are basically sequel Lite databases that are spatially aware so those are really cool you can actually query those in place and do things with that but a geo package is basically a packaged sequel Lite database that knows what a lot long is so to speak but the the most common one you'll run into if you're dealing with geospatial data are shape files and shape files are basically Evans just nodding along in pain I see that pain brother shape files are basically the Internet Explorer of geospatial data they've been here forever everyone uses them they're ubiquitous and they bring you nothing but pain but it's what you get and the data that I've I've built my up around to give some context early is the Kegel Cova 19 data set for confirmed cases globally of Kovan 19 how apropos I know but it's a shapefile it's like okay Kahless isn't sitting this chip shapefile I believe they have it in a couple different formats but the easiest one to grab was those good old shape files fun fact a shape file is not a file shape file is four to five files in a folder hopefully zipped often not you can see the breakdown of a gesture to my second monitor like you can see it a shape file is composed of dot SHP s HX d BF and p rj there's another one that I've forgotten off the top of my head that this is like optional I think but you can also tuck that information in one of the other files it's a good time but those all of those files are necessary they all basically are referenced by that initial dot SHP file so you can't just kind of pluck one out when you parse the code and we'll look at that here in a second when we ingest some a shapefile and kind of look at it for a minute they all it's all required you'll only parse the shapefile the data is SHP but that shape file references data and all those others so they all have to be there it's it's just a good time all around but yeah if you're working with geospatial data get used to shape files you'll you'll you'll be there with them for a while so we talked about getting into those shape files and different things so we're talking when we talk about shape files geospatial data layers is kind of the default term the the layers of the shapefile and that's what we're gonna hop into now but to do that I've got to introduce you to some more jargon and tools specifically judo and orj they are painful but they are very useful and if you've got any experience with them you know exactly what I'm talking about ddos stands for geospatial data abstraction library and it's a sea library that is very fast very efficient thankfully we can interface with it with things like Python but you also can use it from the your bash shell of your tee show or your terminal and I'll show you that in just a minute but that's a great way to get into there and then also orj which fun fact Oh our G stands for literally nothing now once upon a time o RG had a significant meaning Oh Archie now technically stands for simple features library because the meaning of the acronym has been lost but they decided to keep it anyway but simple features library aka o-r-g is a vector library that's basically absorbed into geo and they're highly links now so the way you ingest geospatial data is through layer mapping you go through a layer and you look at it and you find the corresponding data connections or you look at basically what is the schema of that feature and then you create a corresponding feature and we'll do that here in a second now that we are at our fun demo time so come along with me into my terminal and we are going to go first let's let me think where I am I want to go into of a demo cases eight uh we look at the Constituent I actually want to do this instead that's better see this is why you always have to temp the top dogs and try and do stuff like want to be invalid argument you're not pop T in this puppy in this alright just change directory there like a farmer alright so it Clara you might bump your font size system yeah thanks again wait command plus plus plus plus plus perfect so inside that is a shape file that is one shape file with an extra XML I think everything else is is is key to a shape file that's in there so those are the constituent pieces of a shape file know that we're there we'll back out of it and bust out some tools to explore that data but your info why are you complaining this is why we always do you're in the cope a demo directory thank you sir yeah and you've got our key info instead of ogr awesome it's like a code review session it is exactly let's try that again so G R cover demo cases data we want unique shape yeah so we got one layer the the the basic info will go in and introspect that and tell us what our layer is that's one layer we see here it's called a unique geo coded matches and it's point data and we'll go into the geometries of shape files or geospatial data on the other side of this but first let's take a dive into that layer and see what we get and here we are so this are the constituent pieces of the the geospatial data in that shape file the very top is defined as a geometry the feature count is how many features are in that shapefile basically think of features as a single row of data and then you'll see there's things mentioned here like W wkt and all of that will go into as well again lots of jargon but then you'll see underneath it the score match address long label those are basically attributes on that point on that geospatial area so if we were looking at a circle which would be a which would be a polygon and I'll go in a moment obviously like everybody understands conceptually it's a polygon but that's actually a field type within this kind of data so if we were looking at a circle and we wanted to describe the color of that we would say color blue and things like that amongst these fields so those are all descriptors of that point so for the Cova data this is an address that matches that that confirmed case that's the Latin long it's the country of Ord that's the country it's in the rank all of those are data points around that alone so that's basically what all all of the data that's within our excuse me our data set that we're exploring so that is at its basics that goes let's look to one second I want to go to my model file here so basically what I've done and I'll walk back and show you again I've taken this and used it this is the schema of the feature and created a model off of that you see here they're basically correlated I even use the max link and things like that defined in that file you go along and create your model and then here is the geospatial field you'll notice that the model I'm using is not the standard model it's the GIS DB model so you you still sorry about that it's still in your class you're inheriting from models model but your model is this different so you're importing a different models there so that can be confusing if you don't notice the import because that differs but with that model you get all your geo your geometry fields as well so I had picked this one it's it's kind of good to do it gets you familiar with your data if you're in a hurry you can use something called ogr inspect ogr inspect and I'll give you the example here will generate a models file for you and give you the dictionary that it generated on your behalf on your behalf basically here's the mapping and once you've got those model files you can run a load script which I've created here that basically takes that mapping either the one that I've hand mapped here or the one that's in the model file that a gr inspect creates and you point it to the shape file you want to ingest and just run it just call it call it into your your Python shell and run it and it will execute and parse that entire data set for you and stored into your database so that's that's basically layering at a high level now we're gonna have some fun talk about geometry so with those geometries we mentioned we have point data which is a Latin alone and that's what we were examining in this example because we're looking at the point of a confirmed case of kovat 19 there are multi points so if there are a cluster of points that you want to associate with one thing there's a multi-point data type there are polygons which are multiple points that have a linear relationship that basically create a nonstop shape whether it be a circle or square or any number of polygons but it is a solid area so it's referenced like a polygon would be for those who want to have negative space within a polygon they have multi polygons so think of circle versus donut in this case your polygon within the polygon represents the negative space just in case you like polygons inside your polygon then we have line strings which are basically how roads would be created so that's a linear connection between points that does not create a completed polygon the shapes not complete in that regard so that's the basics of geometries that are supported within Geo Jengo there are a lot more geometries available but those are the ones that geo Gengo supports out of the box so some fun facts there is for every every field every geometry you have what's called the srid or spatial reference system identifier and basically there's different ways to kind of interpret the points within that geospatial area particularly when you're looking at geographical stuff when we're looking at geometries just in a purely geometric sense those things aren't relevant we'll get into that just second but say you're looking at a highly you want a very accurate depiction of say a spot in Texas there are SR IDs that are highly trained or highly defined around Texas versus more broad ones so for example if I pick out of here and go and look at our data here we see the SR ID on this one is 43 26 which is basically like the most accurate us like most general USSR ID that's that's the default you get with Gio Gengo and it's most commonly used for like a broad swath of of geometry of geographies within this stuff and then you also have the issue of geometry versus geography because you can store geometries within these things because that's purely that's purely what they are is when you take that geometric geographic element and add things to it that things are computed differently so like distances with geometries are not calculate the the metric the unit of the unit of measurement for geometries but when you are using geographies you're obviously getting either kilometers or miles and you can specify that as well as geography is measured on a spherical plane geometries are measured on a Cartesian plane so and then as our IDs as well can can kind of differ like if you're doing like Flat Earth Maps know Flat Earth jokes here if you're looking at if you're calculating your distances on a flat earth map you you would use a different srid then if you are looking at spherical distances or globes all of those like when you're looking for extreme precision are really important and so you have the ability to get that precise with your geometries so there's different supported return types from geo janggo you can have so if you were to ask for that point back you would by default get what's known as the well-known test text those are supposed to say no not no I apologize typing is hard well-known text which is just a string representation of those Latin long coordinates and then there's the binary representation of that as well which is the well-known binary you can also ask for the extended well-known text or binary which will include the s or ID like most accurate what exactly is this point here it will give you the Latin long and the srid so that if you're concerned about getting conflicting data that said there's using a different srid than the data you stored you can cast and things like that and the geo jason is another like standard get this out of the box kind of thing and it's again it's it's increasing in popularity and hopefully it becomes kind of the de facto but we'll see what we get now it's time more demos y'all because everyone loves to watch me try and code in front of you all so we're gonna take a look at a few things in the demos we're gonna look at admin because admin gives you a ton of stuff out of the box we're gonna look at rest GIS rest framework of course is everywhere if you're a django developer you gotta have an api and there's g is aware library for for that and then we'll we'll see if we can come up with some fun queries to run and i didn't actually have this on here because I wasn't sure that I was going to be able to get it working but I've actually got a friend and slapped on it too pretty stoked about that so we'll kind of go through all of that stuff and that concludes the presentation portion now you all get to watch me share my screen and futz around with files things of that nature so jumping back over here we looked at the model file we've looked at my mouse is the worst so I talked about admin being great to to bring in the geospatial aware admin all you have to do is inherit it make it a hit from geo model admin and you'll get a ton of stuff out of the box so let's take a look at that [Music] hurt your eyes don't look at the API yet we're not to the API yet alright for an admin so the case is here in admin hmm or that data said I parsed you can go in here and look at all of these so let's look at Ramshaw so here's the data that we got off of that based on that mapping to the RAM jaws in Minnesota in the u.s. it constants it's technically in a village or is a village it's got a score of 71 and I honestly don't recall what the score is but you can see there's string representations of the XY that are varying degrees of precision those came in that way the the Django admins kind of obfuscating a little of that for us but then you'll see here this is and this is the the neat part you'd a built-in map to display your point right there in admin if we zoom out we'll see we're in Minnesota that guy is pretty far north and then one of the fun tools you get out of the box is to say like no that's not where Ramshaw is I don't saw that we're here I'm gonna say good point save it I'm good we've redefined that that Latin long with a madman it's a powerful tool so be careful with it if it were a polygon I'm sure you saw here look at all of France that's good France County France in France is it is France dead center of France so there's a polygon editing tool here if this were a polygon instead of a point you could draw a polygon and that would be saved to the database as well so pretty cool stuff you get just straight out of the box with Django admin jumping back over to the code let's see here Oh serializers we talked about restaurant where GIS are you have a geo feature model serializing you pass that in and inherit from that be sure to specify what your geo field is and then specify what else feels you want to include in that serializer and that will generate valid geo jason to you instantly that's super helpful to connect to your front ends which we'll see in a minute obviously you'll need to connect it to a view for this one I've added some additional stuff here you know obviously just normal view set stuff but more specifically I've added a bounding box filter that will be used to the rest API so that you can send a bounding box which a bounding box is just a polygon like I want to look at everything within these these points basically creating a radius by which to search against and that can work and then this data set was like five gigs of data and Ida Ida scoped it down so I've done that but I've got a the full view that I hope to be able to show you and not crash my zoom meeting with the big case for you in a little bit without that slice but to do where else did I need to go or want to go rather pull one last file see your my demo breasts have talked about so I'll show you the rest API real quick as I foots around with there we are so you've seen admin now we'll go over to the API maiya cases and here is that truncated Wuhan of course scores the highest typical Wuhan and you have allergy audition here this is what it looks like as a feature with the geometry at the type point and your coordinates and then all the properties that that go along there just in case we can't see that always bigger always bigger hopefully that's helpful for anybody you need to see it it can't but yeah that's good that singing screen so that's what a defined point of our data in geo Jason looks like all your standard queries would work ranked and things like that like you search by all of your your basic properties but then you can also and I'm gonna do this on I'm gonna do it on big cases because don't come back because otherwise it won't one sec it doesn't like you you can't filter when you've sliced and I've sliced for the sake of I tried to use the pagination there's a pagination told for it couldn't get that working in the midst of my scrambling to get a demo app completely feature complete before this demo so it is what it is but now I'm gonna hit this in point with the bounding box bring this back over and so now we've scoped to that bounding box so you see we're in Mississippi now we're no longer looking at China so that's a great way to take from your front-end should a user through their JavaScript create a bounding box on the front end and want to limit the data that way it's real responsive I mean you guys saw how quick that was so it works really well and then you've also got or RM queries that you can utilize and we'll i'll show you can some kind of examples of those I tried to find like the geographical bounds of some places like Bentonville and Fayetteville and places like that but it's hard to just kind of google and find it the like give me the G the geography that long of this place if you know the way to get that please please tell me cuz that would be fun to play with when you find it it's almost definitely going to be in shape file format so I wasn't able to find a nice bounding box per se beyond that for us to write into an ORM query but so I'll pull up some example Docs here that I've kind of hashed out and made my own so that we won't run them in the terminal I apologize but you'll get the basic principle so you define a point if you wanted to you can just make instances of these classes obviously so a GIS Geo's point and you put in a Latin long and then you can say get polygon by polygon contains point and so you would be filtering by you'd be filtering the polygons by those that contain a specified point or in this case with our code we would do cases and we can do this the inverse of this because the syntax allows you to specify with the ORM queries you specify the field and then there's a lookup there's a list of spatial lookups that you get and so for this one our field is called point so it'd be get point and then we're using the within lookup because we would want since we're looking at point data we would say give me the point within this boundary or this polygon give me all the points within this polygon so that's how an ORM this would or it would be a filter not a get because there'll be more than one guilt er yeah so that's how that would work as far as those go and then additionally you can do distance queries so you can say give me the point for a case with a point of distance less than and then you specify the point you want for comparison and set that distance like for this one it's 7 K this women to be 7 K as well just without the kilometers so you can look at distances differently that way so again that's that's not gonna run in my console so sorry we can try and do that but I think it'll be more fun to try and watch my my Dockers blow up when I try and ingest all the datasets of this thing and put it on a map so we'll do that next let me see so yeah looking at how we would Institute a front end just in Django because you there there are a lot of tools for visualizing data it's kind of what I started off with was talking about how everybody loves visualize geospatial data one way to do that that's got out just got almost out of the box support for Django is using leaflet so Oh Evan Evans the man that's great I got a fine I'll look at that a minute so you would use the leaflet so this is a django template that I've created that creates a leaflet map you've got some leaflet j/s I you would install leaflet pip install included in your install apps and then you can use it instantly I went straight to the CDN version of these libraries but you can npm install them and do that if that's your preference and so down here is where obviously is I say obviously down here's where I render the map leaflet map I'm calling it us I'm giving a callback method the is defined here and that function excuse me is is a standard one takes the map and the options I've specified where the API is that I've created in in geo Jason or geo Django no rest framework that's the one and then it makes that call populates it on the front end so if we look here we have a nice map I call it us because I centered it on Kansas which is I looked for the geographical center of the US and they gave me that so yeah we are we'll back it up wonder I there my zooms probably and you set your zoom in your center within the settings that PI file for leaflet so that when it loads still be there but then you can you can see that this actually also has the information the United States that's a poorly named data entry but you can see New Mexico here that's deer run number one Colonia so the fig I'm guessing that's a city in North Carolina to run some of these are at the state level I parsed in the proper name I believe is the field let me see what did I tell it place name place name that's it yep so you've got those on all of those points and if we we hop out I limited this data set to 100 so that's 100 points off of that data set on a map and I did that work probably in an hour maybe the hardest part was forgetting to put the dot jason at the end of my url because it tripped everything up when i didn't specify that in case he pointed that out to me so he got some some coming to the rescue points there and that helped a lot I don't know working was like it doesn't III could get it in the DOM and I was look and say like it's pulled it into the Dom its rendered this is 200 the data is there why isn't it displaying but all the JavaScript wanted my URL to end with da j s instead of just the view set so lesson learned there but the good news is also freed it up to work on another tool so this is leaflet leaves is great and it works really well with django to make something more robust though there's a tool called deck GL and along that is another tool called Kepler deck GL is a is a point rendering tool written completely in JavaScript and web assembly more specifically or more accurately not JavaScript so it can render a ton of data points Kepler is a tool that sits on top of deck GL and gives you a lot of power and so what we're going to do now is ingest there's there's that same data set here in Kepler we've got an instance of Kepler running and one of our kubernetes clusters so that looks pretty cool you can change some stuff you can zoom in but the real power of Kepler is with a lot more data so what we're gonna do now is try and melt my computer by hitting that big data endpoint and everybody loves Big Data right so let's see how long this will take and what color the smoke it is um to forget your dot Jason's see he's too good to me so I'm here for he you all you all need a colleague and boss like Casey Kinsey willing to push that top button my mouse my mouse battery went dead five minutes ago but I'm still pulling through in there like a champ so we're gonna let that pull out of the back end of my little computer here but it should be around sometime in the near future all we're waiting I have a quick question I'm not kidding okay so the first two one view and Django is essentially a controller right correct yeah okay I wanted it was really dumb and they said hey everybody's calling that thing a model view controller you know what's cooler than that MTV right love the real world yeah so it's MTV in Django and decide the other the other is some I'm currently in rail slammed right now and I was curious does Django do something kind of neat like where you within a single function you can specify Tim templates I guess in the Django sense based off of the type like you just appended to dot JSON to that call did you have to do something in your routing to say go here if my type is JSON not in that one but it doesn't have the response to like like like rails does that's one thing I miss as in former rails developer you don't have any response to where you can dodge a synergy CSV there's not that kind of magic in in in Django but the dot Jason works and we're all help we're all really happy for you do have that kind of magic in PRF do you and erm yeah DRF does handle that for us you have it like you can pass a format query string parameter DRF or you can do a Dutch Jason but it's not we yeah it's not quite as easy I like the controller level though where you can like very easily trigger that logic but but you can pass it through and access it on that on the code side oh I didn't have my network up to see for was too late talking about Tyrell don't give up on it it's pythons not very good at serializing a hundred thousand rows of Jason I'm giving her all she's got captain my best Scotty it's not very good I apologize but you can definitely hear my my fans kicking on on this bad boy sort of MacBook Pro it is indeed 16 15-inch 17 these are things that 16 is the biggest now does it have the Escape key the physical escape key it does not okay so it's the 1515 man it's cooking the visual is so worth it the payoff is great if we can get there it's always it is is it okay rest call tech that long or was it Keppler ingesting it mostly yeah on my sad little machine trying to do all that I think that long to cook the cookie yeah we did something similar on a project using this exact stack with decked out on top of it and something like this that many rows it's all Jason serialization and pies Jason serialization this week's been turning it to geo Jason we you can do Jason serialization with post GIS it gives you functions for that and this same type of thing you can get down to like sub to second yeah do it indirectly in the query so that says that's a future optimization if you need to to render this kind of volume of data unless what we've done on the other app was we wrote that all in sequel just for the Jason rendering for the front-end when you're dealing with this many points of data because I mean that's a lot as you can tell let me see Casey you're better at playing with Kepler than I am what do I need to do to make this thing really cool make it look you want you look really badass yeah yeah open that thing up open that layer and then leave it as geo jason and then underneath the color do color based on and then select some some numeric field from the geo jason score rank what we think but i don't know that i their phone you rang that'll be interesting you'll see how they're ranked but how that plays out geographically and then pick a more robust color palette something I think they have some ones with broader colors or pick a divergent one where you've got like oh yeah really highly divergent color yeah you pick two opposing colors with a neutral in the middle I think it's the third grouping of colors oh I see what you're saying i ones as dissonant yeah if you go all the way down you should get like a red to green yeah go up to two bars from there you want this guy oh yeah so you're lowering stuffs can be red your high ranked stuffs gonna be in green so all of that data pops up there of course that all comes back to the esoterics of that data set and what value I don't know what they're calling rank and that data set but right but there you go that's a make it look cool that's how you get that VC money all next up profit who wants to go in I'll sell you this out for like 3 grand right now docker eyes that send it to you sit here radius it to your cousin it works as a lawyer in DC got the cove in nineteen data set right in there you're ready to go look at the the radius based on and set the radius based on score they may be highly corner now they're not correlated otherwise yeah no you're right as your bigger ones would all be the same color I'd be interested in know what the difference is there then it's better when the radiuses are smaller yeah so couple of not only fun tool for playing with this kind of data and you can build very comparable stuff using deck GL and we use view as our JavaScript front-end of choice so we've built we've replicated a lot of these features with you and deck GL and matte box for the underlying mapping so though that's but again that's all rendered from the little Django app I've got running in a docker box on my local machine so there's definitely a lot of power there and again like the there's a ramp up time with just kind of understanding all the esoterics of the information but once you get in them the technology is there to help you move pretty quickly so that's what I got and I'm glad to chat about any of it and yeah thanks for listening thanks for coming to my TED talk as they say well thanks Tyrell can we give you like a is it weird to give you like a virtual round of applause it's very weird everybody clap into your microphones I can't because I'm using push-to-talk and I have my hand on my spacebar ok awesome you guys Tyrell you want to do you got some time to do some questions if anyone for sure all right lay it on it have you looked at the John John Hopkins data set that map that dashboard they have yes do you think they're using something similar they had I don't know if they're like because it's an open source thing and I think they're using arc juice events as room that's that ESRI tool ok yeah it's witches aren't just are just you know it's the same one that you it's the same - boring - like if you go to the city of Fayetteville maze there's a jury tools for like mapping out board boundaries and like the trail system it's all on ArcGIS that's a really cool yeah rgeous is in the shape file of the same company just as the tool and ESRI also made the shapefile yeah exactly so there's a lot of prior art there and data and all of that so it gets heavily utilized well Bobby's got oh nice they were fairly John Hopkins does their stuff on get so you can fully that's pretty cool I don't care if it's not in shapefile format that's better I'd rather parse it out of a CSV then touch a shapefile yeah it's so tools like Kepler require your stuff being a JSON file a geo jason rather than shapefile so thankfully with with rest rest framework GIS we get gyeo Jason right out of the app so you just at the end point in here and your app there and it pulls it right in my question is when was someone gonna tell me that I spelled my own name wrong in the chat today pretty hello kynzie pretty bad Casey kind that's why I'm wearing my hat I'm incognito Dino you Tyrell I had I was gonna thaw Siwon and then I saw my own name and I completely lost my train of thought that's why I'll come I'll come back if it comes back around and I'd like to eventually clean this up a bit and make it available on the lofty github account so that folks can pull it down and play with it on their own there's some there's some noise in here that I would want to get rid of and flailing and things like that but eventually I'd want to do a I'll do a SlideShare and get you guys all this stuff so that all you have to do is docker docker compose up and have yourself a geo tool to play with but have can you maybe just elaborate a little bit on what you're able to do with like with decks so deck GL is a is a framework that runs on the GPU to render like highly graphical map stuff Kepler is a data exploration tool that you just showed us that's built with deck GL so you use the deck GL framework and then implemented tools to like look at datasets and change colors based on values um can you maybe just elaborate a little bit on like what you can do with layering and you know adding also polygons to with like deck geo like what kind of visualizations can you do with that beyond putting points on a map sure so actually here's their this is fun now in 3d mode oh you know you're not scared anymore oh wow that's a disappointing so we kind of explore further with the tool we have here I've put it in 3d mode and if I chose to let me see if I can extrude height based on school maybe rather than radius I'm being pretty sure I'm pretty sure Kepler's extrude tool only lets you apply it to radius Polly's I don't think you can do it on point fields name alright so there we're better we beat Kepler deal with it so the identity of the deck GL framework allows you to do a lot of stuff in regards to adding like extra dimensions three dimensions to your data so for example you can set a height based on a value so you can you can and then set a corresponding color based on that too so there's really amazing ways to kind of look at what's going on in a region to where you can get in there and see a lot of detail and communicate some really powerful stuff and that's not and with even within I'm trying to think I've seen it play out oh no no I'm sorry so you one thing we've done the kind of represents granular stuff within a larger context is we create a polygon that that covers an area and then we take data within that polygon and break that into smaller subsections of that data basically we grate a polygon so that there's data within each of those grids that conveys meaning within that context of that polygons and then we can extrude we can color and introduce attributes like height within that polygon so it's one thing to just have like stuff at a poly a single polygon level that gives you a meaning but when you can within that polygon have subsets of data that that also change it becomes really powerful and so you can use your external polygon to the finer region and set some boundaries and some lines that are the boundaries but then also on top of that because it renders in layers you render additional layers in there that show meaningful data at a more succinct geographical space that that can give you a lot of meaning about one polygon whether it's comparing polygons so when you can stack data like that you'd also stack points within those polygons so that you can look at all of the stuff within a specific region so creating bounding boxes kind of out the gate are really great as well is that kind of yep you're kind of hitting on the stuff you want me to cover yeah I mean you can like you can stack them in layers which lets you let me do all kinds I mean you can show people maybe even just toggling the visibility of that layer in Kepler off and on the little eyeball on the left side up there where the purple mark is this guy yeah so you can see how quickly Dec GL I mean that's rendering a hundred thousand points off and on like that so if you can imagine that's just one layer so now imagine building a UI that stacks multiple layers on top of each other that you can flip off and on that quickly and you can start to imagine the ability to you know overlay datasets you can set transparency based on values and so right you can put transparent layers on top of satellite data so you can see the underlying typography underneath it and then there's like a whole bunch of that we haven't even messed with and any of our products but animation so the deck Jia was open stores icon by uber yeah an uber can animate a billion cab rides in New York City and draw out every route and play it with this software which is really really cool and it's a know the way it does that is it basically it creates creates those layers and then rather than destroying them and putting them back it toggles the visibility on and off so and that's that's the beauty of being able to write all that webassembly is to be able to do those kinds of things keep them around per system change them visually without having to repaint the dog like well I mean technically it would be rate repainting the Dom this is where I'm gonna get over my skis talking JavaScript and Dom but without having to recreate things in the Dom simply toggling they toggling them off visually you see Bobby says can I change how many points or groupings based on how soon in to limit so you can do clustering that'd be that'd be clustering and I believe it may have some clustering built in we were having to we were considering making our own clustering at some point and we weren't finding the clustering tools we were looking for originally but there's I believe there's some clustering in pyramiding okay I had I've not heard that term so it's quite possible I'll Google that shortly my screen he's the word pyramid for for when there's a like a broad overall view and then if you zoom into one particular section instead of seeing all like see you've got nine sections that make up one layer of a pyramid to look at at the generalized less data passed over the internet version you're looking at one layer but you can drill down into layers deeper so I'm it's sort of similar yeah sounds very sim now that you're not looking at my screen I'm gonna research that word Theriot if you let me I'll I'll brag on you and share my screen I pulled up I think I've got the right thing here but that is that is very true Bobby so this is a product can you guys see this it's looking sweet yeah this is a product that Tyrells and and Tyrell and his team here at lofty have been working on for one of our customers kind of a less abstract more practical application it's in farming so you can see here that they're using this is using deck GL embedded in our view app with Jango on the backend and it's using polygons here to represent the boundaries of individual farms and to be able to see that again satellite imagery it becomes really clear what this data is representing right that that's a strong visual tool and I'm trying to see if this is the right data set I'm looking to see if if we have more layers on this there's uh this is our staging environment so I'm just trying to pull it up I don't know if you know Tyrell which one of these projects actually has some goods no I don't often it's been a while since I've been in there yeah we got a bunch I know we do everything a while back yeah there's a lofty southern hang that sounds like something I would make it does it's in this one if I can find the right one can show some of that 3d extrusion things like them I'm not trying to hijack your Q&A though oh yeah this is the one this is the this is the money right here all right cool so let's pull up corn as a crop agreed yeah corn date is usually more dense right yeah so let's look at their yield by elevation it's like information looks flat though that's not what I was expecting to see how meticulous was it to draw out all of these polygons uh that's a great question honestly the my understanding of this data is that the farmers themselves usually have this data and I think they go in and they bring them in yeah too bad I know that this is our staging environment so it's got some stuff on it this is this is in progress normally what you can see it you can kind of see it when I when I close the opacity down a little bit you can see that grid tessellation almost in here they're little quarter-acre grids within these fields normally they're colored it looks like the color stealing is not doing it correctly but yeah so you know you're drawing boundary of the fields and then it's representing by height the elevation and by color the yield on those fields and so there's a lot of really cool stuff you can do with it we're if you've listened to our podcast lately you know that we're pretty high on deck jail and have been for about three months now or more and the to see kind of this stuff in case he's talking about or I send the link to it are our sizzle reel man our oh that's a good idea that's a real the sizzle reel is got some sand we captured video of it we were sparring I and I said god that looks like a really good at I'm gonna get a video of it just in case we break it and staging and we can't show it off anymore so you can look at it on YouTube instead that's what we'll do you have some epic music - it's so great you're gonna go be ready to fight with William Wallace when you're doing watching that thing but yeah it shows kind of the power of Dec GL and some other some of the visualization it's just a it's a promo reel for the agency that we work at but it does show the power of Dec GL and specifically the data that the Casey was trying to demonstrate that we broke in staging his developers tend to do doesn't matter stage that means we're shipping code for guys they might have any questions for Terrell this was pretty awesome I hadn't had a look at GIS from from a programming standpoint but I've had some training in GIS what context was that him the the local community college in the in in California and also I worked with it in one of my previous careers okay des ray ArcGIS yeah QGIS is another big tool okay you just installed on this Mac yeah that's that's that's one so that if you place them at the Erdos imagine no I haven't led with that one mm-hm they used to be pretty good I don't know what they're like now but I saw this do they still do have a tool so what's been interesting for us we find ourselves as an agency pretty consistently in this niche of helping data scientists or people who have kind of taken on that role within their company replicates some of the there's some really powerful open-source tools around geospatial data or other kinds of data-driven things these days right and so it allows for people to come up with a unique idea and a unique concept and unique way to build data or work with data which is exciting but then they don't through those tools have the means to take that and offer that at scale to a customer base and so what happens is we often have business people come to us who are a conglomeration of like a couple businesspeople or even like a full business that has it has some guys who do have some skunk work stuff and they're saying like we've found this thing but we we can't scale it we've got a guy with like three jupiter notebooks and qgs on his computer and we know that he's got like great information but this has got a scale and so those are the problems that we find ourselves being asked to solve and pretty excited to solve these days but yeah but the problem is is we end up having to be competitive with tools like Q gists and Kepler and all these things there's like well I mean I just using Kepler like this what can you think do what Kepler does he's like we don't have the budget of uber bro to collaborate we literally had a client ask us when we were building a geo feature they had asked for so I don't remem remember what they had specifically asked but we looked at it it was gonna be really ambitious some very interactive thing on the map they're like you just does it is open-source just go get it it's like now yeah a thousand engineers that are way smarter than us working on it for 15 years uber does it how much it cost do something like that be a deck of corn then you can do this yeah so I don't know how much it costs but I'll tell you how much they had to spend [Laughter] cool okay well thanks everybody for coming out in and hang in for another remote meetup we'll keep doing them this way and tell you know you know you're all living it so that's the plan for now and so far they've been good I know it's kind of maybe awkward a little bit different than the normal but these have been a lot of fun so far so we'll look forward to do another one sounds like next month we're gonna be doing a definitely a dot file swap maybe a dot file talk to kick it off as well yeah you're not familiar with that concept then totally show up change your life yeah so that'll be next month and in the meantime you guys thanks for hanging out and stay safe and we'll see you next time and I'll I'll post the video of this recording probably on YouTube on unlock C's YouTube channel and then we'll share the link back to the meetup group so you can share it or rewatch it or anything like that with Tyrell you might send your slides out and post them on there or you know you can get them in the video so yeah I'll get the slides out soon and then hopefully have the code base up on github here in the next like shooting for within a week so we'll see it was a very cool Tyrell thank you very much I could have you seen hey everybody take care guys we'll talk to you soon

Channel: Lofty

Views: 4,312

Keywords:

Id: 6B7VZuq51ew

Length: 71min 51sec (4311 seconds)

Published: Fri Apr 17 2020