GeoEvent Server: An Introduction

good evening everyone my name is r.j. Sunderman and sitting here with me is cigar IRA we are product engineers on the real-time geo event server product team and this session is an introduction to gu of nth server so thank you for joining us as the last question for your day quick look at the agenda for this session we're going to be defining the term real-time GIS what does it mean when we say things like event record or real-time analytic saeger is going to be taking a look at working with real-time data to a demonstration making features come alive I'm going to be discussing an introduction with applying real-time analytics there's a deeper dive into real-time analytics coming up on Thursday then fagor will be discussing a little bit about stream services integrating real-time data with web apps and wrapping up there should be about 10 minutes when we're done if you have questions geo event server at ten five is not an extension of ArcGIS server it is something that extends or enhances the capabilities of your ArcGIS Enterprise it is real-time for your data Enterprise it has five key capabilities the first is making it easy to ingest data from real-time data feed such as JSON feeds or XML feeds all you have to do is configure an out-of-the-box connector rather than write code to try and bring in these data feeds once you've brought in these event records into the geo event server we want to be able to perform continuous analytics on the data we want to be able to filter and process the event records as they are received and that's what we mean when we say real-time analytics geo event server extends or enhances the ArcGIS data store as a component of your enterprise with a no sequel database an elasticsearch spatial-temporal big data store it enables you to store larger volumes of data at higher velocity than you might otherwise be able to using a real a tional database such as Oracle or sequel server once you have begun to accumulate large volumes of data using another functional server geo analytics you can do batch analytics on your data geo event also provides the ability to visualize high velocity and high volume data using on-the-fly aggregation and finally notify those who need to know about patterns of interest as they occur I'd like to take a look at three of these that we're going to examine in this session and then invite you to visit us over at the real-time island if you would like demonstrations of capabilities such as the on-the-fly aggregation or the spatial temporal big data store so what is real-time GIS I think we're all familiar with the concept of a feature record and the fact that you bring attributes that have location associated with them into a geo database in order to expose through a map or feature service the challenge is that with the growing number of sensors out there in the world what with sensor networks and social media feeds vehicles that are reporting the GPS location the data is almost out of date as soon as it is loaded into your enterprise you need a way of connecting up to a real-time data feed to bring the data in as it is produced a couple of examples of real-time data I mentioned automated vehicle location or vehicles that are supplying their GPS coordinates police cars ambulances any vehicles that are in a commercial fleet probably here stateside AVL is the largest single adopter in the Geo event space but we can also track individuals that are walking around with phones or radios so now we're dealing with personnel safety and making sure that we know where the people are and that they are where they're supposed to be sensor networks measuring temperature pressure humidity we could have a sensor networks spread throughout this building with 500 different sensors that are giving us environmental controls and readings in real-time so that we know if people reporting that over in smoke tree it is getting warm it's because the sensors in that area of the building are 10 degrees out of Tolerance and there's a problem with the HVAC at that end of the conference center you could also attach RFID tags to critical equipment if you wanted to track where certain life-saving equipment was as it moves around a hospital for example lots of different examples of where we might be receiving real-time data another key capability that we're looking at is the ability to do some sort of analytics on that data such as detecting spatial coincidence in this slide what I'm discussing is the concept that we've loaded a geofence an area that fishing vessels are not supposed to enter into the geo event server and then as we receive a is reports or GPS reports from the different fishing vessels we can detect spatial coincidence inside/outside intersects with that geofence in order to display meaningful information such as alerts speaking of alerts we have the ability with geo event server not only to add and update features in a geo database but to push out SMS text or email alerts because we don't want to be slaves to a web map waiting for a green light to turn red or vice versa we want to be alerted when something is happening that drives us to go look at the web map in order to do an analytic and determine a an appropriate response in this slide what we're saying is that a bracelet that is providing a location say like on a child if that bracelet has left the school area during a normal school day we want an SMS alert to go out to a parent so that they know their child is left school so those are the five core capabilities of geo event server bringing in real-time data in a configurable manner without having to write code like I say we're not going to be looking specifically at the spatial temporal data store or geo analytics some of those capabilities if you're interested please come visit us over at the island with that I'd like to hand the presentation over to cigar is going to talk a little bit about making features come alive thank you RJ given silver ingest real-time streaming data from a wide variety of sources using the input connectors there are number of out-of-the-box connectors which are provided with geo vent as you can see here on the left the ability to pull an algae is our four features put an external website for Jason geo Jason or XML you can also receive features geo Jason Jason or a rest endpoint or maybe on the web socket you can also receive RSS feeds or receive text on a TCP or UDP socket or subscribe to an external web socket for tation or geo Jason if you have your data in the form of a flat file or maybe CSV geo JSON or JSON format you can also have go and look for a folder so those were all out-of-the-box connectors as you can see in the center column we have a lot of community based connectors which are available on the ESRI gallery their enterprise messaging applications like activemq message rabbitmq CAFTA so we have connectors for that cap which is a common alerting protocol which is used in emergency applications enemy 0 1 h3 which is a GPS standard so if you have a device that is meeting its enemy a 0 1 a 3 you can have that data into geo when you also have connectors for social media Instagram connector and Twitter connector as well as connectors for here Wireless and symbol you also work with a zoo partners who have developed connectors to meet their specific requirements we develop Amazon IOT and azure IOT connectors let's say if you want to bring your scalar later you have the OSI soft connector or if you want to take a look at the 250,000 vessels across the world on an SD map you can use the exacto T is connector you can also have data from your city using Waze into an ESRI map using the base connector the connectors in the center column have their associated data projects so you can fault them as well in addition there's also do when Java based SDK so you can create connector of your own if these connectors listed here do not meet your requirement similar to the input connectors you also have output connectors so geo and server disseminate real time and analytic results using output connectors the number of out-of-the-box output connectors with genius as you can with your one server as you can see here on the left you can add or update a feature or publish text to a UDP socket you can push this in geo Jason to an external website or web socket you can send text message SMS email or instant message you can also send features to a stream service you can also write to a CSV file or geo tation or JSON file let's say if you have high volume high velocity data and you are interested in working with the spatial temporal big data store you can use joven to adder of data feature we also have connectors on the ESRI gallery for Amazon IOT azure IOT and enterprise messaging applications like ActiveMQ rabbitmq CAFTA you can also write to your Hadoop or MongoDB you also have a connector for Twitter similar to input connectors these connectors also have their associated data project so you can follow those and create a connector or expand based on your requirement if any of these connectors listed here to not meet your requirement there's a java-based SDK so you can create your own connector as well now that you have seen about the input and output connector next we will take a look at how to apply real-time analytics to streaming data for that you need to create a duo and service with which essentially defines the flow of your event data between your inputs and outputs you have your filters and processes winters can be filters or spatial filters as you can see over here the data is coming on a TCP port data in this case is a vehicle data and you have four outputs we also have three processes over here you're trying to monitor the speed of the vehicle let's say for example if the vehicle is speeding you can get an alert you can also monitor the dangerous area for example which is essentially geofencing let's say if and if the vehicle that you are interested enters or exits a particular geofence we can get an alert we can also get an alert in a third processor let's say for example if the vehicle is at a particular distance or at a particular time you can get an alert for that and a lot can be in the form of an SMS or an email how do you work with real-time data in ArcGIS let's take a look at high-level steps for making features come alive the first step in this process is to connect the output to your feature service or maybe the stream service next you will import the Geo and definition now what are the German definition it is essentially the schema of your underlying incoming data a German definition essentially let the Jew and server know what the structure of the incoming data is it includes their acid attributes and the data types next you need to configure your input to connect to your real-time streaming data now that your input and your output you need to connect your input to your output using a given service and finally you're going to visualize your features on a client application like maybe an Operations dashboard or a web map so let's move on to the first demo of the of the session making features come alive in this demo we are going to take a look on how you can bring your Waze data on an SD map through a g20 meant to do that I need to log into geo and manager now German manager is very similar to obvious lower manager you can create your inputs your outputs and geo and services as well as you can perform administrative tasks like importing and exporting geo and configuration you can also import and administer geo fences you can also import and export geo and definitions as well so the first step as I mentioned in this process is to create an output so I'm going to send features to a stream service in this case so I'm going to create a stream service output connector so these are the properties of stream service output connector I'm going to name this as a screen service out and I'll cut it to the default our CI server I am going to select the given definition wave alert I will take a look at the definition in a minute I'm going to publish a stream service and display field name will be UUID that's the track ID in this case and the geometry is point so stream service is published and I will say if they connected now let's take a look at the Geo event definition that I use in this case I use the wave hello geo and definition the German definition essentially consists of the name which is the attribute and the type what kind of is it an integer string geometry or timestamp cardinality and if is it track ID or geometry let's take a look at the steam service that just got published so now get to server rest as you seen service which are essentially the same field that includes the location and UUID as well let go back to geo and manager now the next step in this process is to create an input I am going to receive data from waves so we have a base connector for gol so I'm interested only in the alert that is traffic incidents from bees and the default area of interest that comes out with the connector is around the city of Los Angeles so that's the polygon I'm going to change the frequency to 30 seconds so every 30 seconds Gio vent is going to look for update I'll save the connector now that you have your input in your output you need to create a Geo event service that is essentially going to connect your input to your output so service designer is a model video type experience wherein you can drag and drop your input your outputs and your filters and processors as well so I can drop my input and output that I just created and I will connect it and I will publish the gon service let's go back to the mitered page now you can see out here that we have input we have output as well as 0 and service let's take a look at the map you can see that inventor flowing to eight eight five events in and two eight eight five events out let's go back and take a look at these events on a map so this is around La so maybe let's take a look at a feature it's about in the city of Carson jam and jam moderate traffic and the street as well so you can essentially create some application with this data coming into G of n now RJ will continue with filtering and processing part of G of X or what you are so what I would like to do now is take a brief look at applying real-time analytics then filtering and processing on the events as they arrive is one of the core capabilities that you want to use you event for a filter is something that discards event records based on an expression only event records that satisfy the expression are allowed to pass through the filter in this first example you see that there is no filter and as sagar showed all of the events are flowing from the input to the output you can filter based on an attribute expression such as is snowplow you can filter based on spatial such as is in dangerous area you can combine both the attribute and spatial expressions within a filter to look for snowplows that are in dangerous areas so I'd like to take a look at that what it looks like in a demonstration in my Geo event server here I have a simulator that I'm going to use to simulate some vehicle reports I've created a couple of different services with inputs and outputs and I've turned on the outputs that are going to send event records out to a geo database to add an update feature records if we simulate a few of these records as soon as the web map refreshes I'm going to have to fix that demo would you please go ahead and call it the slide deck and continue with the next section actually hold on one second sorry about that I have a filter actually in place that was discussing all of the data that's why nothing was showing up on the map so here I have the ability to send data into geo event extension you can see that some of those vehicles began to move forward if I turn on my legend here you'll see that a larger diamond is used for the vehicles latest position that's an update a feature with some smaller dots showing a vehicle history report so as I click the simulator to begin to continue sending data the web map of refresh and the trucks move forward on their routes if I take a look at filtering now I can come over into my Geo event service and rewire the event processing so here's a filter that is looking for a vehicle type of snowplow that's a literal string one of the string attributes and the data that I'm receiving all I do is include the filter in the event processing event processing routes click publish and now when I come over to my map you'll see that only that gold diamond that that snowplow is moving forward so the blue truck is not moving and as I failed in the initial part of the demo there were no dots displaying because I was filtering all of the data that was coming out I can look at a different filter if I want taking out the filter for just is snowplows and the second filter is combining logical expressions for looking pickup trucks that are in a high-risk zone the high-risk zone is that geofence that you saw over on the right-hand portion of the web map I publish the changes that you event service and simulate some data you'll notice the gap in the pickup truck because several events were discarded from that earlier filtering but now the snow cloud is not appear to be moving forward and the pickup truck is because it is both inside the high-risk area and it is a type pickup truck we use processors when you want to calculate new values in rich event records projected geometry anything that's going to change the event record that you're working on a couple of common processors might be a field calculator a simple field calculator can convert meters over two miles or concatenate two strings together field calculator accepts a configurable expression you can actually invoke some of the Java string functions that provide some very powerful capabilities to do regular expression pattern-matching and replacement and replacement when working with strings a field mapper is responsible for changing the structure of an event definition often the json or xml data that you're going to be receiving could be hierarchical and complicated in nature you need to simplify that structure before sending the data to a stream service or to a feature service to add an update feature records a field in richer doesn't attribute join on an external table in order to enrich the event record that's being processed with information out of that table if we're dealing with fixed sensors for example that are not moved around there probably not broadcasting a location but in order to see them on a map you need a location so you're going to enrich from an external table to bring that location that geometry information into the event record so that you can display it on a web map a geotag ER works with geofences to identify spatial coincidence such as inside or outside and enriches the event with the name of the geofence that the event record shares a commonality with an incident detector is one of the monitors that uses a filter expression when that filter expression is true such as vehicle outside area or vehicle speed greater than 50 can open up a new incident record and begin tracking the duration of that incident finally track gap detector is another monitor if you have configured track gap detector to say I expect vehicle reports every two minutes and it has been five minutes since I've heard from a particular vehicle or from a predict particular staff member on their phone then new records are generated by the track gap detector to alert you to the fact that data that was expected is missing taking a look at some of these processors in a demonstration I'd like to look at incident detector and track gap detector for this I'm going to open up a new web map just so that I don't have the old data on the screen and I'm going to come over to my geo event manager and I'm going to turn off the inputs that I was using for the earlier demo and turn on this demo number two you'll see here toward the bottom of my screen that I have some notification outputs email and SMS text I'm not going to include those in this demonstration since it's going to be difficult for you to hear my phone when it chimes and I've received a text message and we don't keep our email open during presentations but if you'd like to see some of the notification possibilities again please stop by the island I'll be happy to demonstrate this for you with those services running take a quick look at what the service looks like we have a couple of incident detectors the track gap detector and multiple different outputs that are using add and update feature records to display the data on a web map when I simulate a few data records we should see alerts popping up over here in the dangerous area I can zoom in and show that if I were to even start playing the simulation those alert records start to move along with the truck as it moves through the area I can click on one of the alerts that's the vehicles latest report but here is the incident that the vehicle is passing through a steep zone area and it's been ongoing for the last 17 seconds in demo time if I send a few more data records and allow that alert to move forward when I click to identify you'll see that it is tracking that is been ongoing for the last 38 seconds I'm going to zoom out a little bit because I'd like to take a look at another set of data that is included here you'll see that this vehicle has been detected as speeding that's a different alert and a different incident detector in my real-time analytic it is a pickup truck that's its latest location its alert is speeding and has been going on for the last 39 seconds if I just pause for a moment and don't simulate any data the track gap detector is going to start providing me some alerts because there's nothing coming in from the event simulator therefore expected record data is not being Araya is not being received and that's what the track app detector is looking for so there are my alerts if I click on one of those there's the latest truck report but here there's a track gap that started tonight at 5:59 p.m. because a track was detected is true on either truck number four truck number five so you can see the incident detector and the track gap detector use filter expressions just like filters but they are processors and they are generating new data based on the detection that they were configured to look for spacial operators used by both filters and processors detect things such as a point inside a geofence but they can also keep track of state we don't necessarily want to know every time that a vehicle's report is inside still inside still inside the area that has been assigned to we want to know when it has entered and in order to do that we need to track States so that we know the last time we observe the vehicle it was outside all of the java geometry operations that are supported through that java geometry api are supported within geo event server inside enter intersects touches we can detect all of these relationships between an events geometry and a geofence events might not be reporting points they might reporting polyline that's where we can start to look at whether a poly line intersects or crosses the geofence geofences themselves while often areas don't have to be you might import geo fences from a feature service for example that are points representing critical facilities to take a look at that as an animation here we have a storm that is approaching the east coast of the United States and we've buffered that storm so that we have a little bit of early warning we've determined that that buffer intersects one of the points geo fences and so we're going to generate an alert and send an email message inside that email message we were able to encode a URL that allows the user to click bring off a web map and quickly inspect to see what the storm is doing and which of the critical facilities are about to be impacted quickly I'd like to take a look at any versus all and how you select geo fences in your filter expressions so you'll see a pair of regular expressions they're dot star slash dot star that is selecting a set of geo fences to consider for this analytic and has nothing to do with the NE versus the all keywords the dot star on the first says any category the second says geo fences of any name the first filter expression geometry intersects all says I want the events geometry to intersect all of the geo senses at the same time obviously they would have to overlap in order for that to be true at any point geometry disjoint any on the other hand is often true because if you consider geofences say as County boundaries or US States just because a vehicle is inside Arizona doesn't mean that it is not disjoint from California Nevada New Mexico and every other state in the Union what does this look like in terms of a graphic intersects any geofence here we have a track line with several track points from a vehicle that is reporting its position the vehicle is found to intersect any geofence because at one time it intersects a and one later B and later C but as I was saying if you want to intersect all geofences the geo offensives would have to overlap because there's only one track point on this illustration that intersects all three geo fences simultaneously as an animation we might have a construction zone that we have sensed off and there's different zones within that construction site that we want to start generating alerts when a vehicle leaves at the find zone so you can see we have some blue alerts popping up here on the screen as the different tractors are leaving the zone that they were intended operate in this tractor off to the right has left the construction site entirely and is disjoint from all the geo fences he should not be driving up this road so that's a different class of alert that we want to highlight with that I'd like to hand the presentation over to Sagar to discuss stream services and integrating with your web Maps in qrj so street services are a way to visualize your yield and data on the map it is an alternative to feature turbine both are useful but they have some key differences well feature services are designed to visualize data in a database now database can be a sequel or not equal database so every time you send to me when you are trying to look to a key term it's going to be a pole from the database on the other hand if you have steam services they take out the middleman just to speak maybe your application subscribes to the service and the data is pushed across to the client so a radar is essentially not archive and it is directly pulled by the client so which means faster with visualization long story short if you want to archive your data for analysis you may use future service if you want to take a look at your data and a much faster way maybe stream service is the answer so although steam services fit inside the LGS platform steam services are something which is on the server side the client applications displays steam services using javascript api object which is called a stream layer a senior is something which is on the client side with that we wrap up with the session object Enterprise is the fully featured mapping and analytics platform given server brings real-time capabilities to the RCS platform and the new licensing roles for the arcgis server which is a component of the object Enterprise allows it to be licensed in different ways so you can use your gon server as your real can server now geo and server ingest real-time data using the input connectors it processes and filters them and then sends it to output connectors it enables continuous analytics to be executed on immediate as it arrived you can also add an update features to your spatial temporal biggest or it also works with gon pick server to make batch analytics on high volumes of data it also provides solutions for high volume data visualization with on-the-fly aggregation using the spatial Abril big data store and of course it alerts key stakeholders using notifications you also have lot of useful resources available for those folks who are new I'll say the introduction tutorial which is a thorough step-by-step guide in steam service tutorial as well it also has a video tutorial for steam services if you are interested in high volume high velocity data and this pressure temporal big Edisto tutorial I think should be very useful notification tutorial gives you an idea of a follower thing we also have blogs and discussions when the community discusses about how they are trying to solve the challenges using geo and server these tutorials are free to download and they come with simulated data so you do not need to have production data you can use the similar data for testing your own application creating and testing your applications they are video recordings of these technical workshops available so geo and server in the past conferences you can take a look at that video with the link provided here a lot of sessions leads it to real time and big data they're going around this conference they were session this morning on Internet of Things which will be repeated again on Friday yes the two sessions for connecting for creating connectors and processors using the SDK I think this should be pretty useful for the deaf community out here Adam Milankovic the power toner for real time and big data is going to present a session for real time and bigger the GS at a massive scale you'll also present a session for best practices so you will get to those tips and tricks of configuring gon server is also user presentation on Thursday for building Android based applications using CTO and server and on Friday you have making tiny things come alive which is a really nice session we highly appreciate your feedback and I must say you can download the application and there's the app is even tap and search for the this session and provide your valuable feedback with that be open for enough questions

Channel: Esri Events

Views: 4,572

Rating: 5 out of 5

Keywords: Esri, ArcGIS, GIS, Esri Events, real-time information, GIS data, IT infrastructure, processing and analysis, streaming data

Id: 6OyHXecQKFI

Length: 40min 17sec (2417 seconds)

Published: Wed Mar 29 2017