Elastic Maps Application

all right hi everyone welcome welcome to today's meetup my name is dahlia and i work for alaska's community team um today we're here to hear from milos who is an education engineer um in elastic and he's going to talk to us about elastic maps application um we're going to take all of the questions in the end so um yeah i hope you enjoyed go ahead milos xdalia thanks for a brief introduction i hope everyone is able to hear me and see me well uh so before we start for today i'm just going to briefly introduce myself and my background so as dalia mentioned i am an education engineer at elastic with the company for almost two years now and uh on on daily basis i'm teaching students and users of elasticstack on different topics such as the stack itself but also different solutions and how uh users can derive the most value from different applications inside of the stack my background is in software engineering i used to work as as a back-end engineer for different startup companies and interestingly enough i started using elastic search uh in 2015 while we were still on version 1.5 and then in the past few years working for different companies have been able to use the stack for different use cases today's topic will be elastic maps application we're going to focus a bit more on uh how we can visualize geo data so we're going to cover a few different topics there first we're going to start with the question that always arise so what is elastic what is elastic stack what is elastic search when i first started working for elastic i used to explain to people that i have a new job and i'm working as an education engineer for elastic and then it took me usually about five minutes until they would stop me and say wait wait wait you work for elasticsearch and then the whole story about explaining the differences and so on so we're going to briefly touch on that and then we're going to immediately move into elastic maps app we're going to look at few demo maps we're going to understand what is necessary for us in order to visualize the data but then also we're going to briefly touch on how do we get the data as we know oftentimes getting the data and preparing the data is the toughest part of the job so where do we acquire the data how do we enrich the data how do we prepare the data so that elastic maps application can work with it effortlessly we're going to touch as well on elastic map service and understand what supports elastic maps application and then after let's say 10 minutes or 15 minutes of talking about stuff we're going to do one big demo and create one map ourselves and make it as nice to the eye as possible so first things first what is elastic elastic is a company this is elastic all these people uh who gathered once a year when it's not covered in the same place somewhere in the world and work for a week together uh in an internal event and discuss things create strategies and decide where the company is going to move in the following years uh elastic as a company is uh behind elastic stack and we're going to see a bit more on the elastic stack in a bit but before we move on to the technicalities it's it's very important for me to share just a bit on the company because i think it's a very unique place to work in and of course as you know elastic is uh behind the elastic stack which is open source and it's a distributed company so there are around 2 000 employees 2 000 of my colleagues at the moment that work from 40 different countries in the world and we all work remotely and we used to work remotely even before we were forced by the current situation so we tend to follow the philosophy of the product internally in the company we have a distributed product and we also are a distributed team and we take a lot care about our source code so the same way as we have the source code for the products which is publicly available so as you know you can just go to github and check the code for elasticsearch or for kibana in the same way we have a source code for the company which shares with everyone what are the values of the company and it is a great place to work in and uh it is it is it's very nice for me to be able to share this with everyone and uh say how how nice it is to to work with colleagues that always push you to [Music] become better to learn more and then of course also spend time privately and do on work on your own stuff so let's talk a bit more about elasticstack and give you a bit of an introduction on where do we find those maps how do those maps work well elasticstack consists out of four different products so we have kibana elasticsearch beats and logstash elasticsearch is the core of the stack oftentimes people confuse the elastic search with elastic but elasticsearch is a product that is at the core of the stack we like to refer to as elastic stack and elasticsearch is in charge of storing the data querying the data running aggregations and it's the brain of the whole stack kibana connects directly to elasticsearch and it is the place where we can find different applications that send requests to elasticsearch receive responses back and then render all these things inside of the ui through kibana you can manage the stack you can discover different data you can create graphs and of course you can create maps as we're going to see today and beats and logstash are those two products that actually allow us to additionally enrich or transform the data before it enters elasticsearch so that it can be used effortlessly now of course uh elasticstack is a tool and a lot of users a lot of community members understood that this tool can be used in many different ways for that reason you're going to hear a lot about elastic enterprise search elastic observability elastic security so different solutions that are based on the elastic stack but of course extend its usability and focus a bit more on specific use cases and elastic stack can be deployed anywhere so you can of course just pull the code from github and run it locally you can download it pre-packaged but you can also run it in the elastic cloud or use tools such as elastic cloud enterprise and elastic cloud on kubernetes to orchestrate the deployment yourself now the question is where are maps in this whole story well maps are a part of kibana but as we mentioned before kibana can't do anything on its own until it's connected to elasticsearch so you need to have at least one instance or a node of elasticsearch and then you need to connect kibana to that node and only then you will be able to access the interface and inside of that interface find maps app so what is elastic maps app this is a simple simple visualization or representation of the map without any layers on it but of course let's look a bit back in the history of maps and geo search and geopoint and generally geographical data have been a part of elasticsearch since a long time ago so even if we look all the way back to version 0.9.1 we can find there that it was the first mention of actually using something like geopoints and querying these geopoints in order to find maybe all the locations that are within a certain radius from a reference point or draw a polygon and find all of the documents inside in fact that was the main reason how i personally got involved with elasticsearch five years ago i was working for a small startup company and we had the need to index documents that represented news articles and then we needed to query those news articles millions of times of course for millions of users but then also do that many times per minute so every time a user or their phone would change the location we would have to query for hyperlocal news and we were looking for a solution and elasticsearch even back then of course offered that for us to index some geo points and then run these different queries now from version 7.3 so if we fast forward all the way to version 7.3 elastic maps application is uh has general availability so even before then i believe since 6.7 it was still in beta version so you were able to use it but it's generally available since version 7.3 and you do need to have a license for it but you need to have at least a basic license and basic licenses for free so of course that means that you can just download it get the license and use it you cannot find it in open source version and you cannot find it for example in amazon elasticsearch service because uh they are not under basic license now these uh that application elastic map service is powered by elastic maps service these are two separate things so of course we have elastic maps application and elastic map service and we're going to touch on that in just a bit and why do we even discuss this today well because we have vast number of use cases for using maps security or seam can be one of them oftentimes you have events as we like to refer to to uh the the things that happen that represent a moment in time that have typically source or destination and we want to use maps to plot these events and to maybe uh look for anomalies or look for potential sources or of threats uh if we look at observability uh any service that you have any website anything that you use to monitor or observe your system you might want to know where your users are coming from how often are they accessing your service from which cities from which ip addresses from which countries and so on there are many many numerous use cases of course to be able to see uh your data plotted on a map in near real time and react to it filter results and so on so let's let's uh look at some maps now i'm going to stop sharing this presentation for a second and you should be able in a few moments to see my kibana so here we are in kibana and this is the home page so let's go and check out some maps uh how do we do that well we have here add sample data so kibana allows us to import some of the sample data sets together with visualizations pre-loaded maps dashboards and so on so i've done just that i pre-loaded three different data sets and i'm going to go and find this maps application in kibana it's enough for me to open the left hand side menu and choose inside of kibana maps application there are already three pre-loaded maps here and one of them that i use as a test reference so i'm going to open for example logs maps and if we open this map we're going to zoom out here and we're going to start showing some data now we must not forget one thing that is always uh first thing that you need to check everywhere in kibana and this is the time picker oftentimes you're going to be tricked into thinking well i don't have any data something doesn't work but first thing that we need to do is say okay let me look at and then i'm going to say here five years worth of data to be on the safe side and now the data starts to appear here so as we zoom in and out of the maps we can see that we have requests coming by country and total requests and bytes for each of these moments and as we zoom in you can notice that these circles start becoming more granular as you zoom out they're becoming less granular so this is just one type of a layer that we can create inside of maps we're not going to dig too deep into this because we're going to create it ourselves when we start from an empty map but as you can see as i zoom in inside of city i get to the point where i can find actual requests so another interesting thing as i zoom out actual requests disappear so i don't need to see them because they would take too much space but as i zoom in to a given city kansas city in this case actual request starts start appearing i can see the details of the request just by clicking on the actual dot and i can see client ip timestamp so some of the fields inside of that particular document this plus sign will allow me to dynamically create a filter so maybe now i want to see all of the requests from client ip79156 so i'm going to click on this filter and this will dynamically create filter inside of my maps application now uh let's go back and after seeing some maps let's discuss about the data so of course maps application will allow us to render the data that we have inside but where does this data come from well in elasticsearch we have this concept called index and indices allow us to store the data that shares common fields that shares common grouping and kibana allows us to create something called an index pattern that is simply a pointer to one or more multi multiple indices that contain some data and these maps work with index patterns so i want to understand the source of the data for this particular map we're using logs total requests and bytes so i'm going to go back to the menu and i'm going to choose here uh stack management and i want to see which kind of index patterns i have inside one of these index patterns is kibana sample data logs and this is exactly the index pattern that has been used in order to provide the source of the data for the map if i extend to the further rows i'm going to see okay geodot coordinates field is of type geopoints so this is what i really need in order to represent some data inside of my maps so let me just quickly go back to the slides and let's discuss a bit more on uh what kind of data you need to have in order to use elastic maps application so when we talk about uh data sources or data that we need in order to use elastic maps application we mentioned index pattern index and mappings now of course uh data needs to be inside of an index and you need to create an index pattern for that index inside of kibana but even more so data needs to be properly modeled or ingested to elasticsearch and geopoint data type is something that you will need if you want to represent coordinates inside of a map now this is of course not the only option you can also use uh actual names of countries regions or cities in conjunction with elastic map service which is something that we're going to see in a moment another option is to simply upload your data in case if it's in csv or geojson format so you don't necessarily need to uh create geopoints by indexing data in different ways you can just upload csv upload geojson and then instruct to elasticsearch how to transform the data in a format of geopoints to be used by geo queries and of course elastic maps and in the end an alternative option is to use pipelines so what if you don't have any location information inside of your documents but for example you have an ip address which is a common scenario in case you have some web access logs data and so on you can use this geo gip processor and you can instruct to elasticsearch to intercept every document that contains an ip address to take that document and to check the ip address and to derive the location coordinates country name city name from the ip address and insert it to the source of your documents in that way you can now work with geo points again and represent data on elastic map service so let's do a brief uh presentation of that so what do i need to do in order to start using maps i'm going to jump again and quickly share my other screen here so if we go back to our kibana i'm going to go to dev tools and inside of dev tools i am going to create a simple index and this index is going to contain just one field of type geo points so let me do that i'm going to put index 001 with type of the fields geopoint and now if you want to index some location data we can do it in multiple different ways i can start with geopoint as an object and simply index a document that contains location fields and then lat and lon shortened for latitude and longitude with the appropriate coordinates i can simply go and index these documents i can also index the geo point as a string so it doesn't have to be an object it can just be a string with comma separated value where the first element represents latitude and second element represents longitude i can also index it as a geohash so bear in mind this is all the same field the same mapping but i'm just providing different source of data so here's the geohash here's the geopoint as an array so i have an array of values like the longitude and i can also use geopoint as wkt point primitive and index that as well how do we make sure all of these things worked well of course i expect now to have an index with five documents and these five documents should all have their own coordinates latitude and longitude so i'm going to perform a simple query and i'm going to use the query type of geo bounding box this type of a query allow you to define a box with only two coordinates so top left and bottom right this will create a rectangle and you want to return all of the documents that contain geo points within that rectangle so i'm going to perform this search request and i'm going to get five results back so all of my five initial documents were valid so this is what you need to do in order to prepare data you need to have at least one field if you want to use if you want to display the actual document on the map that is of type geopoints but even in case you don't have those geopoint fields you will be able to interact with the maps by just using the actual values of fields that represent country names uh or postcodes or region names and then join that's together with elastic map service so uh let's quickly show you the elastic map service we were talking about you can access it by just going to maps.elastic.com and this is how elastic map service looks like now elastic map service is something that allows you to actually use uh base map tiles to use shape files layers and then provide layers on top of that it is developed and maintained by elastic so separately of course from the elastic stack which also means that uh you don't have to wait for the next update of elastic stack or you don't need to update your whole stack if you want to get the freshest version of elastic map service they update separately from the stack itself and they're based on open street maps okay so this is the base of elastic map service we also use natural earth as well for this service and what does this service provide to us well if we zoom to through this map we can already see that we have different levels of zoom so we start with the world map but then as we zoom in further we can go all the way down to street level and then also see inside valuable data for particular locations so if i zoom in to amsterdam which is my current location here i'm going to be able to see street names restaurant names museums all of the points of interest on my map now on top of that uh you have three options to choose tile layers so the default one is this uh light map there's also an option with roadmap dark and the roadmap desaturated that does not contain all of the information that you have on a regular roadmap interesting thing inside of this elastic map service is vector layers so for you can either choose world countries but also for different countries you can find different vector layers so if we look at the netherlands here we're going to find provinces of netherlands and if i zoom in now to the netherlands i am going to see these vector maps inside and i'm also going to be able to find with which values these provinces are uh represented with okay now netherlands might not be a good example because we can for example go to the u.s where we have counties states and zip codes so if i choose usa states i can see that for each state i have name fips five to alpha code so two code abbreviation or um us dash again to code a two letter abbreviation why is this important for us it is important because in case we don't have geopoints and we just have fields that contain values for state name or two-letter abbreviation or italy provinces we can join these documents together with vector layers in elastic map service and this is exactly what we're going to do in our demo um okay i'm going to quickly move to making a map actually so let's do that let's start with creating our own map and work through all of these theoretical things that we mentioned so far so i'm going to share my screen again and let's go back to kibana and i'm going to go get out of dev tools and go to maps application now inside of the maps application uh let's go and create our own map so we're going to start from an empty canvas and then add things inside let's make sure the time picker is fine we've got last five years worth of data we have our base layer so loaded from elastic map service as you can see in the bottom right corner using openstreetmap contributions okay as i zoom in and in elastic map service you have up to 24 levels of zoom i can go as detailed as i want into any place and in the bottom right corner you can also see zoom level so here we went all the way down to 24. okay top right corner maybe you can't see as it's really really small i'm not sure how the screen is rendered on the zoom webinar but this is the biggest zoom we can have inside of the maps application so let me start with adding some layers so on top of this base map i want to add layers and i have multiple options here upload the geojson that we talked about before but then also using my indices or index patterns from elasticsearch documents or choropleth clusters and grids heat map or points to point so let me start with point point-to-point and requirement for point-to-point layer is that you have at least two geo points inside of your index mapping we're not able to join different fields from different indices so you need to have in one single index at least two geo points so i'm going to choose my test index for today which is mail delivery and what we should do before jumping into maps every time is we should quickly check in discover section how does our index pattern behave what kind of fields we have inside and if i go here and check my json formatting i see i have location which is of geopoint so geo location and then i also have user geolocation so i do have those two geo points i also have some region names usa states which is going to turn out uh useful a bit later when we create the map so i'm going to jump into maps application again and create this map and add my first layer point to point i'm choosing an index pattern mail delivery source so let's visualize how our users where our users are sending the packages from or mail and where are they sending the mail to so user geo.location is going to be my source and my destination is going to be geolocation and as soon as i do this we can see now many many lines appearing from any given point here on the map to any other point on the map i can add this layer i'm going to name it my sender to receiver layer i want this layer to appear on all zoom levels i'm using aggregation of simple type count and i'm saving and closing this layer now of course this is not really useful now because we're displaying all of the customers who are sending all of their packages to all of the locations so very very confusing and ideally how we're going to use this is we're going to use it by writing some kind of queries inside the search box so maybe i'm going to say id of a user or in this case let's say um user.age should be greater than um or equal to 30. so now i only want to see which users older than 30 or maybe let's say 50 years are sending packages or mail okay for a moment i'm going to click on this layer and i'm going to hide it i don't want to display it just yet and i want to create another layer on top of my maps i'm clicking on adding a layer and this time i want to display the documents themselves so i want to display every piece of mail or every package that has been received at the place of receiving the package so i'm going to choose documents i'm going to choose an index pattern mail delivery i'm going to say use the field geo dot location now here's what happens here we know that our data set here has 900 000 documents or 900 000 packages or pieces of mail that were sent and this is more than 10 000 so kibana here specifically maps application asks us to do three different things either limit results to ten thousand so only show ten thousand results or show top hits per entity which means i'm going to choose the entity and for example that can be geo.country.keywords and i can set it to 50. so that means for each country only show me 50 hits and now as i zoom in to netherlands for example i am only going to see 50 packages destination or mail destination here okay or i can say show clusters when results exceed 10 000 and then as i zoom out now my kibana says well i can't really display all of the things at once here so show me clusters and in these clusters show me how many documents or packages you have inside this dynamically happens so as i zoom in now we have specific documents or packages appearing on the map okay so i'm not going to uh edit like that i'm going to just say i want to have limits results to 10 000 but this is fine because i have this filter here that says dynamically filter for data in the visible map area so as i zoom in further down more dots are going to appear as they get rendered inside of keep on okay i'm adding this layer i'm going to name this second layer as um mail destination and i'm going to leave the zoom levels for 224 for now let's change the icon as well so these are pieces of mail so i can just conveniently choose the icon that represents post and let's leave the color or change the color to another one so let's use a bit of uh it's a darker color here okay perfect and once i did that i'm going to save and close this layer so now we have single pieces of mail displayed on the screen i want to add another layer now and i want to add a heat map now this heat map is going to also request for an index pattern so let me choose the same index pattern made delivery and geo dot location geospatial field as soon as i do that heat map starts appearing and of course it will change dynamically as i zoom out and zoom in to this map so i'm adding the layer i'm going to name it as a destination heat map and aggregation can remain count grid resolution is one of the three options course fine or finest so you will see the difference if i move to finest those parts of heat map actually become much more granular so we're focusing on more narrow area compared to coarse and we can also change the color range i'm going to keep it as the default one okay i'm going to add this layer as well so now we already have two layers and arguably this is becoming a bit too crowded but let's make it a bit more crowded let's add one more layer we talked about documents we talked about using heat map now add let's add a layer of type choropleth so clicking on that i'm finally going to connect some data that i have inside of my index and join it with elastic map service uh vector layers so what is the layer that i want to use if we look back at elastic map service i'm going to say well let's use world countries so these are the boundaries vector layers and what are the options that i have two code abbreviation two letter abbreviation the actual name of the country or three code abbreviation okay let me choose world countries and now what is the join field either two alpha code or three alpha code here i'm going to choose two and now i need to join this vector layer with some data that i have inside of my index pattern what is the index pattern that i want to use well here i have more options because now this type of a layer does not require for me to have a geo point it only requires for me to have a field that has a value that can be joined with a vector layer i'm going to use the same index pattern mail delivery and then what do i want to connect or join with iso alpha 2 code well it has to be geo.country code 2 dot keywords what happens now data is joined with the vector maps and displayed on top of our maps application i'm going to add this layer and i'm going to name it um mail items per country because i want to understand how many packages or pieces of mail are received for each country so i'm going to say mail packages mail items per country tooltip fields we want to keep this one as the field we used for joining so we can later filter on it we can also change the metric we're going to keep it as count here but i can also say well actually let me change the aggregation to average and i want to see the average of packages sent per person or i want to see the average of items in each package i'm going to keep it on count for now and i'm going to change the fill color it's set to by value i can make it solid but that doesn't make too much sense because we actually want the color of each country to be uh in proportion with the number of packages or pieces of mail received and then we can also change um is it going to be as category or as a number and how it is going to be displayed so i can maybe choose a darker color here and now if i zoom out a bit i can see that in the us we have a lot of mail received a lot more compared to other countries for example libya here has less than algeria in europe norway has significantly less than for example united kingdom so i'm going to go and save this layer as well but now arguably we got into a problem called too many layers too many things on one in one place which is not usable for anyone so we have these world map layers vectors that we have as we zoom in actual pieces of male received then we have heat maps and this is not usable so i want to fix that and we're going to fix it in a simple way we don't necessarily need to see everything at the same time so i'm going to argue that when i'm on the world map level i want to see number of uh items received in each country as i zoom in further i want to slowly start seeing the heat maps so i know on which region to focus on and then when i get all the way down to the city or the region i want to see the actual pieces off mail or items so how do i do this well i'm going to first rearrange these layers i'm going to say i want to have mail items per country at the bottom so this is the last this is the first thing that i want to see then i want to have heat map in the middle and then when i get all the way down to the bottom i want to have male destination so specific piece of mail that has been received now of course this is not enough we still have similar problems so we don't see well because everything is overlaid on top of each other but what do i need to do now now i need to change the zoom level so if i go and click on edit this layer it says visibility and zoom levels from 0 to 24 and this is not necessary so i'm going to say well let me see this from 0 to four i'm going to go and save it and then when i get to four i i don't want to see countries anymore i want to see destination for heat map so i'm going to set it four to roughly about ten and then when i get down to 10 i want to see mail destination so exact pieces of mail that have been received as i go save and close this now we can see how the map is much more visible but all the layers are there and we're moving in between the layers as we're zooming in further so now we're at the zoom level of 1.74 as we can see in the top right corner and i'm zooming in so we can already see a lot of mail being received in united states i'm going to remove this tooltip united kingdom france germany of course i'm going to go and zoom in towards europe here because i'm currently there and now we can see this transition between four zoom level where heat map starts appearing and countries disappear so i'm zooming in towards netherlands a bit more heat map of course dynamically changes but then as we get towards amsterdam heat map disappears and i start seeing specific items being received in their exact location and as we know 24 zoom levels so i can if the location is perfectly accurate i can even see which house has received this particular package now i do have to of course mention something this is a fake data set so we're not really working with the real data so no intrusion to anyone's privacy this is generated data set but i believe it's a very good representative of what we can do uh with elastic maps application um i'm going to wrap it up here

Channel: Official Elastic Community

Views: 1,764

Keywords: elastic, elastic maps, kibana, mapping, elastic maps application, geospatial, maps application

Id: Op53zHlH3TI

Length: 42min 18sec (2538 seconds)

Published: Thu Oct 22 2020