ArcGIS: Understanding SDI and Geospatial Infrastructure

hello and welcome to our session on arcgis understanding sdi and geospatial infrastructure in today's session we'll explore how a fully integrated geospatial infrastructure connects you and your partners multi-source data and catalogs you'll learn how arcgis's open platform enables fair data that's findable accessible interoperable and reusable see how sdi collaboratives around the world are putting data to use in amazing multi-organizational geo hubs i'm jill salago simel product manager for sdi and inspire i'm joined by my colleagues bruce harold product manager of arcgis data interoperability and martin hogave senior consultant with esri professional services let's get started i'm one of the founders and former executive director of indiana's statewide sdi in november 2005 rsdi hit a turning point when a tornado ripped through three counties in southern indiana the local emergency management director arrived on the scene pulled out his phone and used google maps for his situational awareness only the maps on his phone were different from the maps other emergency response teams were using local officials were overwhelmed the national guard was called in utility companies and volunteers descended from around the country to help they needed a common operating picture a common sdi open and accessible to everyone regardless of their background or technical savvy 22 people lost their lives that day and that was the day i knew our sdi needed to be available to everyone everywhere because first and foremost sdi are about problem solving addressing shared challenges among multiple organizations from local to global scales people and economies cross borders disasters cross borders nature crosses borders and infrastructure crosses borders our location data and technology must cross borders too shared sdi framework data also known as foundational data provide a geospatial common operating picture for program management and decision making how we think of sdi is evolving today geospatial infrastructure enables new patterns for modern sdi in the 1990s sdi were born out of the recognized need for more institutionalized data sharing in a time of ad hoc and bottom-up data transfer and sneakernet in 1999 our statewide sdi implemented with the help of esri our first nsdi clearinghouse node let me just say our aspirations then were as big as they are today but technology definitely wasn't ready for us yet like many sdi over the years we established our statewide common framework for all units of government today under the leadership of our geographic information officer and gis council it aggregates local authoritative data from 92 counties into seamless statewide coverages and delivers high resolution imagery over 94 000 square kilometers to nearly 7 million residents but it's been a long and arduous path to sdi development and of course sdi aren't static they are constantly evolving you may have experienced a similar journey or perhaps you're just getting started i feel quite optimistic about what this decade will bring i like to think of it as technology finally catching up with our aspirations of course making this happen isn't just about data and technology a holistic framework is required to implement an integrated geospatial infrastructure and put it to use effectively geospatial infrastructure is integrated it connects organizations across borders jurisdictions and sectors so you can address today's challenges to do so we need to use the best science design and systems thinking geospatial technology integrates these approaches as well as multiple data types and allows us to look at problems holistically experience from sdi around the world shows that there are a few major components needed for modern sdi to thrive governance data and technology and people through community engagement and capacity building this aligns with the un ggim integrated geospatial information framework for modern sdi and geospatial collaboratives at all levels of government each of the four components break out into sub-components for example governance includes representative organizational structures leadership and vision strategy and more this year we're building a resource library to help connect you with the resources and good practices in each of these areas the internet and cloud computing are transforming the way organizations manage data and collaborate web gis is significantly easier to use deploy and integrate into an sdi ecosystem than traditional systems users can now inexpensively and efficiently access immense amounts of geographic data web services stream content to users based on the permissions given by the data owners this is the evolution of sdi i referred to it is enabled by an emerging geospatial infrastructure next let's look at some examples of effective sdi sharing portals and while they're all anchored in data and technology let's consider how they also support stakeholders needs beyond technology most people don't rally behind data they rally behind goals and aspirations organizations like we see here in loudoun county virginia are having great success making their data accessible by featuring their programs and initiatives through decision-ready information products and loudoun county's data catalog powered by standards-based human and machine-readable metadata is available to other discovery portals tools automate these data flows behind the scenes with little intervention for example loudon geohub automatically produces a d-cap feed for their curated catalog of data here we see how the catalog flows from the county to the usdata.gov catalog arcgis hub supports dcat which is a resource description framework or rdf vocabulary designed to facilitate interoperability between data catalogs published on the web data are made accessible to all stakeholders first through consumable maps and apps second through apis downloadable data and view services and third through federated catalogs if california were a country it would be the 37th most populated country and fifth largest economy as of 2020 at just over a year old the california state geoportal has 32 contributing partner organizations with five coming soon serving over 2 000 items in their geoportal the geoportal success has been called an anti-silo data sharing effort that made the portal become a reality as it continues to make incremental improvements keeping your authoritative data current is essential to a well-used sdi with over a thousand image and feature services and 137 web mapping applications maryland is keeping it fresh with their data set freshness dashboard each data set on maryland's open data portal is updated regularly their dashboard shows how many data sets have been updated or have not been updated recently enough the dashboard itself is updated on a daily basis in an sdi the stakeholders are the geospatial community of interest and initiatives are often about collaborative data we see this in sdis all over the world like what we see here in washington dc and in north carolina with nc one map nc one map is a far-reaching project that exemplifies horizontal and vertical collaboration nc one map promotes a vision for geospatial data standards and best practices data currency maintenance accessibility and documentation and a statewide gis inventory it comprises 37 priority data themes including statewide ortho imagery and aggregated parcel data from all 100 counties in north carolina plus the lands of the eastern band of cherokee indians with geospatial infrastructure existing data can be combined in new ways and supplemented with additional business data to address local to global challenges the united nations sustainable development goals hub delivered data and services that put actionable open data to work and sdgs are not limited to countries for example the state of hawaii is doing some really interesting work on their sdg hub with the aloha challenge ireland's another great example of build ones used many times in practice with their award-winning geohive geohive has dozens of applications and initiatives that address issues from the environment to the economy and when they need it that principle billed once used many times can be put into practice quickly by leveraging solutions new application portals can be built quickly and easily using existing data and templates like ireland and hundreds of others around the world did to communicate vital covet 19 information to the public and decision makers i think one of the most remarkable geospatial success stories this past year was the scalable deployment of applications within just days and weeks we saw the deployment of hundreds of covid19 dashboards to countries all over the world that pattern has been repeated with the rapid deployment of dozens of other coveted 19 and vaccination distribution analysis workflows data and applications it has raised expectations for the reusable solution deployment on a scale the world hadn't experienced before it's really quite remarkable because that would have hardly been possible just a few years ago without the geospatial infrastructure to support it as more and more actionable data are available it opens new possibilities to put those data to use today we have hundreds of solutions that help solve common business needs through integrated workflows focused maps and apps this is important these are freely available resources that sdis can build on to keep their portals relevant as they mature arcgis solutions leverage an organization's authoritative data and are designed to improve operations provide new insight and enhance services in government utility public safety telecommunications and commercial business now that we've seen some really wonderful examples let's look at the technology behind the scenes sdi and geospatial infrastructure is inherently multi-organizational geospatial infrastructure interconnects multiple organizations in an integrated system of systems federated catalogs and portals expose actionable web services to put to use in applications that make an impact interoperability is achieved using multiple patterns tools and standards these aspects are best understood in the context of arcgis as an open system open means many things to many people here's a way of looking at it well it's always about being interoperable an open system should also try to embrace many perspectives for some it's about being extensible customizable and flexible for others it means standard support open source and open apis and for others it's about enterprise integration and pragmatic interoperability esri remains committed to an open design architecture that enables our users to work more easily within a heterogeneous it environment across many types of data sources and applications from a product perspective arcgis content and solutions provide resources that can supplement your sdi data for example and i think this is really powerful arcgis living atlas now provides continuously updated live open street map feature data that you can use freely in your sdi this isn't just the openstreetmap basemaps you already know and love these are actionable feature data extracted from osm and ready for use in your maps and apps arcgis solutions continue to add new solutions like the 3d base map solutions that help you get more out of the data you already have your multi-source data are managed and integrated using arcgis pro and enterprise and here's where i want to highlight two products we'll see more of in our demos arcgis data interoperability and esri geoportal server arcgis online is where you collaborate with partners and publish web services in online your shared data are highly scalable performant and secure and they come they comply with open standards and specifications to assure interoperability many organizations do this using a hybrid implementation with arcgis enterprise arcgis hub is your delivery and engagement system where you share open and secure content through catalogs that help make your data fair open data catalogs can also be delivered through arcgis enterprise sites together arcgis is an integrated solution for modern sdi this summer we're releasing a new arcgis inspire open data solution that follows this streamlined implementation pattern the new solution represents a lightweight approach for the european inspire directive and supports the psi2 open data directive that goes into force this week it can be used in arcgis online only and online enterprise hybrid implementations enabling arcgis with inspire and alternative encodings and d-cad ap is important because it enables our european public sector customers to leverage the full capabilities of arcgis online and hub while meeting their inspire and open data obligations sometimes the data you need to expose from external providers may use slightly different data models or are already available as feature services next bruce is going to show us some workflows using arcgis data interoperability that can help you integrate and update data from multiple sources as the framework of your geospatial infrastructure bruce thank you jill and hello everybody we're going to take a look now at an actual sdi the inspired geoportal self-service access point for countries in the european union where there's a vast array of thematic data in in gis forms let's drill down now into the country specific map here and even further into the country of luxembourg or more correctly the grand duchy of luxembourg we can see that we have i think we see 304 data sets available of which the vast majority are downloadable and a lot of them have map viewers like wmts let's say i'm interested in the address data and i can see that there's one of the major themes available is address data and what are my options here well i can i can view um in wmts i believe i can also download so i get hold of this data set here in epic application specific gml so it's a dialect of xml it's not natively consumable by arcgis pro but easily made so with accus data interoperability now the for the purpose of viewing in my map here i have built a couple of tools for getting hold of that data here's a tool which does an initial load reading from that inspired gml file and there's a few themes of data in it like the address point data and therapy and names and zone fields and so forth now i've i won't go through this tool in detail but what it does is pulls out elements from the xml stitches them together into a geo-relational format like we're used to doing in arcgis and writes them out to a feature service my address point feature service so it's pretty simple data i'll quit this tool here now this is what it ends up looking like so here i am in in luxembourg i can see the address point data that i've built like i can query some of the the features in the map go with this guy here now i can see i have the house number i have the thoroughfare name i have the zone field the the neighborhood and the post code plus the country which is in french there so all as well i've got hold of my data now that's gives me a bulk download of the data and all's well i i've made it into feature service it will scale right across the arcgi system but a little bit of a clerical review here i have discovered that in fact the data is maintained on a weekly basis a little bit more frequently than is available through the inspired geoportal i don't get hold of that weekly update of data through this portal here hosted by the luxembourgish data platform people now i can download geojson in this case now yes we can read geojson in arcgios pro but what i want to do is take that new data source on a regular basis and update my existing feature service so i've built a tool that does exactly that i've run it already here today to save a little bit of time and of the original address points there we looked at the map a minute ago 167 000 and change i can see there's a few more available at the url for the geojson data 172 000 so just do a little bit of work in this tool here to get the data into the schema that i want that's going to agree with my feature service here i do a comparison about what features are unknown to the existing feature service and i write those features out to an additional layer in the feature service and you can see we have a little over 9000 features output and i just make a remark in passing here that the schema that i'm adopting here is a simplified version of the inspire schema which we are offering as a basically a flattened way to get hold of the data from inspire and make it immediately useful in arcgis so this we already have 9 000 of those features that have been added to my map back in pro here this these are those extra features they're coming up in green and let's just go into the bookmark where it'll be a bit more visible where the data arrived and i can see in my map here if i turn the extra address features on and off i can see around this locality which happens to be a transport hub in luxembourg i get some extra address points there that i might use to augment a decoding locator built from the total set of all the data so this uh a very simple uh use of a pan-european geoportal called inspire and we're using an additional data source to enhance it so you might find many reasons or many ways in which you're going to do something similar in your environment thank you thank you bruce one of the greater challenges in multi-source data collaboratives is the integration of data from heterogeneous sources after bringing in these data bruce can seamlessly publish to arcgis online and add metadata in this case inspire metadata and share with everyone using arcgis hub when shared with hub users can immediately interact with your data and access it and through downloads and apis speaking of metadata let's step back for just a moment and consider another dimension of the open platform fair and open data which are critical to sdis and geospatial infrastructure your data never stand alone metadata the key information that describes your data live with your data and flow through the platform with it powering search in curated catalogs when you enable metadata in your arcgis online and enterprise organizations all kinds of magic happens to help make your data fair your geospatial data are findable when described richly by your metadata they're accessible in machine readable formats as well as complying with accessibility guidelines your geospatial data are interoperable easily published and shared using international open standards like ogc web services and the emerging ogc api family of standards as well as open geoservices specification your geospatial data are reusable when they're released with clear and accessible data usage licenses geospatial infrastructure is really helping to evolve sdi from being data focused to being user focused putting actionable data to use in decision-ready information products geospatial infrastructure interconnects multiple organizations in a integrated system of systems distributed catalogs from multiple sources can connect using open standards and communication protocol such as ogc's catalog services for the web or csw as we saw with bruce's demo sometimes the data you want to inter-operate come from multiple catalogs including third-party catalogs next martin will share some new ideas for catalog interoperability using esri's open source geoportal server martin thanks jill in the previous demo you saw bruce publish a number of web feature services and metadata for it in this demo i will take that and show you how we can use this in arcgis pro and other client applications what you see here is a metadata record in iso 19115 format for a web feature service hosted in rcs online in the distribution section down below you will recognize the link to the arcgis online feature service i will publish this metadata record to my local geoportal server-based metadata catalog geoportal server is a standards-based metadata catalog that supports multiple standards including iso but also ftdc as is being used in that other metadata record there over the state of kentucky in addition to supporting multiple metadata standards we also support standards-based interfaces based on open search but also based on the otc catalog service csw this case my european capitals wfs service was recognized and the link to the service has been extracted for use in client applications now let's switch to one of those client applications namely rts pro for arcgis pro we have developed this csw client an open source free add-in that can search catalog services of various makes and allow you to consume the data you find there let's start with the local catalog service which happens to point to my csw based on geoportal server not surprisingly i find one item that includes wfs and that is the european capitals layer i can consume this layer directly in arcgis pro but this is interesting but it requires me to set up geoportal server populated with metadata and that do and doing that manually is a lot of work but we have a solution for that with the geoportal server harvester in the geoportal server harvester you have the option to configure a number of input types the source from which you will get metadata and you can define an output broker the target to which you will publish the metadata so in my case i've configured an arcgis portal input broker that happens to be a group in arcgis online notice the group id in this registration part and then i've configured an output broker that happens to be my local geoportal server instance this group in arcgis online includes a number of the items that bruce had created during his demo and some other items that we had created prior to this session and it will harvest those items into my local geoportal for that i will configure a task that takes the metadata from the input and puts the metadata in the output let's run this task this is a small task so we can easily follow its progress on the homepage of the harvester and as you noticed it is already complete if i go back to my geoportal notice that i went from two items to seven items this may illustrate the power of harvesting metadata you can harvest metadata from other catalogs and quickly build a sophisticated and comprehensive list of content that might be of interest now let's see how it shows up in arcgis pro repeating the search for wfs to my local csw now results in three items instead of one including the european capitals wfs item that i had but also two others that i harvested from rxs online i can obviously consume these services again in my rts pro while this is great it basically prevents me from having to upload metadata manually to my geoportal but i still need to set up a geoportal if i want to work with content from arcgis online well there is a solution for that also what we have done is we have configured a csw interface directly on arcgis online and the link to that catalog service is what you see here notice this link includes the organization id that's the organization id of the subscription that we have been using in these demonstrations if i repeat my demo for searching for wfs but searching against the arches the inspire online csw you will notice i get a slightly different set of results but this just did a csw search directly to arcgis online retrieving content from the arctis online subscription we work in and of course i can consume these individual items for example this address luxembourg download service following the inspire specifications notice that there will be some points added to the map in luxembourg and let's zoom to that area these are the address point that bruce found from the luxembourg data download site and then included in his address data layer with that i'm concluding my demo and i'll just repeat very quickly i've showed you geoportal server and how it can ingest standards-based metadata and then expose that metadata through a csw service i then showed you how arcgis pro can search that csw service and consume a web feature service found in the catalog i also showed you that geoportal server allows you to set up a harvesting process that takes metadata from arcgis online into the geoportal server and then expose that content via csw and then lastly i showed you how we can create a csw interface directly on rcs online consume that csw interface into a csw client and then consume that directly into arcgis pro for example where we have developed the csw client thank you for your attention thank you martin federating catalogs using open standards such as csw or dcat really help connect data providers clients and users in a heterogeneous environment geoportal server and data interoperability can be great complements to your sdi i'll conclude with what i consider the most impactful guiding principles in an sdi journey amplify your sdi by leading with initiatives initiatives speak directly to purpose they're the why behind an sdi it's what stakeholders care about most feature some of the projects initiatives or programs that are supported by your efforts we see leading sdi helping stakeholders to answer questions become informed and make decisions with appropriate data and information products sustain your sdi by reciprocating value what is the value proposition to your partners sustainable sdi align vision mission and goals among partner organizations it's organizational interoperability and they return benefits directly to contributing partners value can take many forms such as increasing the visibility of a partner's work sharing customizable app templates and workflows hosting public data and facilitating this for that data exchanges finally embrace open and make your data fair open data are essential to an sdi make your data findable accessible interoperable and reusable in our increasingly interconnected world the work you do is essential these principles are helping make your shared data accessible and pervasive so thank you for joining us in this session we hope it's helped you think about some new patterns for modern sdi links to connect with us in the esri community are provided below on the session page next i invite you to chat with me bruce and martin as well as our colleagues listed as additional experts at the bottom of this session window

Channel: Esri Events

Views: 600

Rating: 5 out of 5

Keywords: Esri, ArcGIS, GIS, Esri Events, Geographic Information System, Spatial data infrastructure, Open data, FAIR, INSPIRE, NSDI

Id: k7LnY6rCT5U

Length: 39min 22sec (2362 seconds)

Published: Wed Sep 15 2021