Google Earth Engine and geemap workshop at GeoPython Conference 2021

hello welcome to the google exchanging workshop at the geopython conference i'm your host of this workshop children woo and my co-host is ko market so in this workshop we are going to talk about google's engine and gmap for interactive mapping and analysis of geospatial big data sets next i'm going to show you how to get access to the workshop material you can go to the url gmap.org and then scroll down to the bottom here click workshops geopython 2021 and then upper right corner here click the download notebook icon and you can download this one to your computer so for example i'm going to download to my downloads directory and just click save now you should have the notebook example on your computer let me dd my other files just to clean up my directory once you have this file and so this is the jupiter notebook for those of you who are not familiar it's basically just a notebook that you can have uh source code and outputs together so you can have one code interactively within the jupiter environment so and next i'm going to show you how to set up the environment so that we can run this notebook and before we get into the details i want to quickly point you to some of the links that might be useful so if you are not familiar with google's engine you can click the link here to go to the google's engine website and so google sending is a cloud computing platform provided by google it's free for resource education and non-profit you can sign up for a google's ending account you just need to click here sign up if you have a edu email then i highly recommend that you use your edu email to sign up use this uh instant approval if you use your gmail sometimes it might take a couple of days to verify uh your identity so sometimes you might not be approved for the account and once you have the account approved then you can come back to follow this uh workshop the next one here is the gmap org so this is the website actually we're looking at right now and the gmat python package is the one i developed during the past year it was uh released i think in march uh 2020 and so during the past year i have added a lot of functions to the package and makes it much easier to use google's engine within the jupyter notebook environment so the units works we don't use some of those many functionalities that i developed for the package you can also check out the link to the google's engine python tutorials on youtube so so far i have developed 55 videos with a total length of 15 hours i also have the other uh course here special data management with google changing also 19 hour 19 videos with nine hours so this worksafe is only 1.5 hours and it's basically a condensed version of some of those video that i recorded before and hopefully we get to know some of the uh useful tools that you can utilize to do cloud computing with google's engine if you have any questions you can click the link here to post questions on github so ask gmap questions on github as you can see from here a lot of people actually uh use this form to ask questions and most of those get resolved pretty quickly so if at any time if run into any issues with gmap or the google earth engine python api you're welcome to post questions here within this discussion forum also the google changing website is very useful it has a lot for example the data set and also documentation so uh if you're new to google's engine you might want to look at the documentation keep that in mind the google os engine documentation most of the uh examples were written using javascript okay so um google's engine has both javascript and python and javascript has both uh you can do uh basically has two components computing and visualization so javascript has both but python most of the functions provided is for computation uh it has very limited functionalities or for visualizing the results or data set and this is why i developed the gmat python package to make it easier for you to utilize uh the jupyter ecosystem that you can visualize the data you can also do all kind of analysis interactively and we're going to um use many many of those tools in this workshop okay so uh next uh besides signed up for the google's engine account you also need to have either anaconda or miniconda on your computer if you don't have either of those i highly recommend you just install miniconda so you can just click the link and from here to download the installer based on your operating system if you are using windows you can download for example any of those um 3.9 or 3.8 either one is fine you can for example click that download the 64 bit then just follow the instruction to install the mini conda on your computer after you install miniconda then on your start menu you should be able to find a menu called anaconda 64-bit and then you can click the icon to just open any contact prop okay so once it is open so next we can create the contact environment so that we can install some packages that we're going to use in this workshop okay so let me go to go back to this notebook and i will show you some of the commands that you need in order to set up the conda environment you can simply just copy and paste and i'm going to show you step by step so contact create we are going to create a an environment named geo and then install the python 3.8 and just hit enter then type y it might take a couple of seconds to download all the packages next we need to activate the contact environment and just you just need to right click so you will automatically paste the command into your command window and hit enter so once you activate the contact environment on the left here you should see this one so geo so is the contact environment that we're going to install the packages next copy this line right click copy and then paste hit enter we're going to install the gmat python package from the contact force channel so this one sometimes might take a couple minutes because it has some dependencies and uses type y because i installed the package before and it's using some of the case on my computer so it's much faster but if you are the first time this might take maybe uh three or four minutes to install all the dependencies once you install gmap we also want to install i'm jupiter notebook extensions because this notebook as you can see here it's a very long notebook and we're going to utilize some of the jupiter notebook extension to navigate the notebook so it will make our work much easier and productive so next just copy this line right click and then come here op right click enter this one is going to install all the jupiter noble extensions and i highly recommend also copy the last line sometimes it might not work correctly so just type enter you should install all the extension and the last one is to install the extension to our user and right click hit enter okay so we have successfully installed all the required packages that we need for this workshop and next um you need to download the notebook to your computer and the job in the notebook so if you have it done have not done so delete the novel icon and then download to any directory you want so in my case uh here i'm a notebook is under my downloads folder so next you can just type city downloads so get into the downloads folder then just type jupyter notebook so this is for windows if you're using uh mac or linux computer you can open your terminal and then similarly just copy and paste those commands to install the packages and set up the contact environment that we need for this workshop okay so next just type jupyter notebook hit enter it's opened jupiter notebook within your browser and let me right click close those steps so next once the jupyter notebook interface is open click the mb extension so this is all the zoopy notebook extensions from here we you can choose some of the extensions that you wanted to use for example uh there is one table of content i mean you can control f and then just table okay you might want to click this one table of contents too and you might also have this one already activate so the voila and the other one might be useful is the high input all and these are some of the common one that you can utilize we're going to use this one table content because the notebook example is very long so it would be nice to have the table of content okay so next go back to files and then just click here the notebook to open the notebook once it is open by default you should be able to see the table content so this is the one that uh coming from the jupiter notebook extension if you don't install uh the extension you might not see this one and we can click this one to saw and hide the uh table content you can also have this one is to hide all the output or all the uh source code right so if you click this one it's going to show you all the source code if you click again it's going to high also the voila so the voila is the package that we're going to use later in this uh no in this workshop so uh but for now this uh just big as it is okay so next let's go back to the top of this uh notebook you can also slightly move this one to the left if you want so once we have all the environments set up then we can start going through the examples but before we get into the details about this workshop i want to keep you a quick overview of google's engine so if you're not familiar with google's engine it's a cloud computing platform and it has a huge data catalog so you can click link here to go to the um the website the google's engineer website so once you're on the web page you can click the data set um google's engine has a huge data catalog over 35 petabytes so you can just here you can type a keywords here and just to search any data set you want for example a deviation or link covering use whether climate data whatever data you want and just click then search you should be to find the data set uh and for my data set that you need also you can go to the documentation to locate some example if you want to know more details about google changing highly recommend that you goes to the documentation webpage although mostly example were developed for javascript but most of the syntax were very similar for both javascript and python so if you click here and then especially this section objects and messages if you want to get into the details about uh how the terminology of google's engine then you need to be familiar with this for example image image collection geometry visual and freeze collection so those are some of the very important one because this workshop is only 1.5 hours we won't be able to get into all the details of this but i highly recommend that you check this out and to learn more about google's engine okay so let's get back to the zoop in the nobel example so we're going to go through here in this example we have a notebook example we have eight uh sections and so we're gonna go through one by one to show you how to get started how to uh basically how to load how to create an interactive map how to uh use google's ng data set and to do visualization and analysis and lastly you can also create some maps for zone publications or or for a technical report you can also export data to your local computer they at the end if you want to publish your data product you can develop an interactive web app to be hosting on your local computer or on a server so this is kind of a work used to the entire workflow for using google's engine and gmap okay so uh let's get started while since we already have the everything set up correctly then we can should be able to go through the normal example so first you need to import the library so in this case we're going to import the uh words engine python api and also gmap we also import the python built in library uh os because when sometimes need to export the data i need to save the data to your computer so all you need to do uh you can press shift enter on your keyboard to execute this while the noble or you can simply just click one so it will be very similar so you can click if you you can either use your mouse or you can use your keyboard and after that we just need to create an interactive map so it's very easy just gmap dot map and then parenthesis keep in mind the map here is uppercase and you can save this one to uh it's available map the reason we're using this is that we want to uh be consistent with the javascript api because the jr script api by default the map is called uppercase map so and then just shift enter you should be to see this interactive map so this is just the base map base view on top of ipyla and on the left here you should be able to have this for example uh this icon to search data i'm going to show you later on the right here you can also have for example the um the toolbar and by the way uh this one looks like this is not the most updated yet so uh maybe let me show you how to actually to update the package to uh to the latest version so if you see for example this one i see the just two columns that means your package is not up to date and what you can do is to for example here select this one and then uh sale or insert insert sale below okay so from here you can just type ge map dot and then just hit tab on your keyboard to bring up this one and then just update and then select update package parenthesis okay and then just shift enter to execute this one so basically this one is trying to download the package from the github repository and then install the latest version to your computer because the the package is being actively developed and some of the new functions that are being added daily so you might not have the latest version but if you run this one you should be able to update to the latest version okay so once you uh execute this one you want to use control slash to comment out this one because you only need to update this once and after that just kernel restart kernel and clear output okay so and then we can run this one one more time one and one create the interactive map so this time you should see this one right now we have three columns of about two bars and they mean this is the most up uh up-to-date version of the g-map okay so as you can see it's pretty simple uh if you want to create an interactive map you can just use gmap.map then you have this map you can also pass you can customize the default map for example uh you can choose the center so by default it's centered around the u.s and the zoom label is four but you are welcome to choose to other locations so this is the latitude and this is the launch team so uh don't be confused sometimes you might be long still like you so it should be uh let long okay and then the zoom level you you can also choose the light mode if you for example here you have too many toolbars you don't like that you can just simply choose this one and then just run you should be able to create an entirety map with just two buttons here zoom zoom out and zoom in and zoom out and then you can use your mouse uh you mouse wheel to scroll to zooming and zoom out the map okay so this is how you can create the uh interactive map you can also add some basement for example uh also just to want to let you know uh because for most examples uh each section we create a map but in in real applications you don't need to create so many maps you just need one map and then you can display at the map at multiple locations so in this case i'm just going to create another map and then i add a basement uh called hybrid so hybrid is a google satellite basement with labels so it's called ivory so you can just hit one as you can see now you can have this uh basement here and you can click here click the the toolbar icon and then the second one here layers so here you can turn the layer on and off okay you can also change the layer opacity if you want you can just slide this this one here you can also turn all the layer on and off if you have many many layers so this is how you can add the basement there are a number of base maps available so if you want to see what kind of basement available you can import the base map module and so once you import the basement my module then you can use the basement module to auto complete so you just type for example base maps dot and then hit press tab on your keyboard it's automatically pop up the list so you can select from the list here select some of the existing base map available in the gmap package so for example i can just select this one open topple map right and then again shift enter it should load the base map onto the map again you can click the layers and then you can turn the layer on and off this is how you can create a map and also add a base map next i'm going to show you here how to change space map without coding so again i'm going to create a new map and here right we click the base map so this is by default we added the google base map but there's a toolbar here if you click this one this map icon and then from here you should be able to select from the drop down list so basically you can change the basement without holding you can select any basement you like for example terrain library okay so the hybrid is the one with google satellite with basement uh labels and you're welcome to select any other one for example a3d relief and [Music] for example there's so many examples you can choose from and if you have any other basement that you like um you can submit a visual request on the gmail github repository i will be happy to incorporate some new base maps into the package besides adding the basement for uh from this existing one you can also customize adding some uh custom wns and xyz tie layers if you want to find some examples you can go to the link here for example the national map here this website has tons of web services you can utilize and users need to click then from here you will see for example wms you will need this one so wms or xyz so if you click any of those for example high zero copy and then open the link you should be able to find this one so wma server but this is the link that you can copy and then come back to here just like this so the first one here the first example this one is the x or x y z tie layer and this one is from uh for example uh google terrain intercept one is create a new one so this one here is the train layer right so the the default one here is the google map but we edited the terrain if you have any xyz time layer from if you know the uil you can just pass the url and then add that one to the map similarly you can this the second one here is wms so at wms layer users need to have the link okay so once you have the link then you can just use the add wms layer to load the data onto the map so this is the one meter resolution net imagery so this is a huge data set and if you already know the wms you just need to load this you don't need to download any data set so similarly you can go to this website to find some other wms services and then you can load that one to the map if you want so uh for example let me find this one here suited usgs ready ready and you need to copy for example this url and come to here you might be just to replace this one paste you also might need to maybe change the um change the names okay so for example swedish redi and then shift enter okay so you should be able to see this one being loaded like this this is the uh at the global scale and you can see the map yeah okay so this is how you can bring in basement layers from the internet from uh you can this can be from a local server it can be from any web server as long as it use the same protocol either xyz or wmsr layers next uh let me show you here is to use the drawing tools because most of the time if you have the map uh sometimes you might need to draw some roi on the map and so the gmat package has some built-in tools data from ipad library so if you create the map and then you can simply draw a rectangle or a polygon on the map or even circle or line so for example so this is toolbar where you can select this and once you for example once you draw those are polygons you can actually have access to the google's engine object once you saw those uh reasons of interest it's being they're being converted to google engine object so they can later use those in your computation so how do you get access to this polygon right you can control slash to uncover this one so the um the object is stored within the xp uh user roi so user underscore roi if you shift enter you should be able to see this one so this is basically a json format and it's not very useful in uh if you're using python if you want to see the real content you always need to use the dot get info parentheses and then ctrl enter you should see this one so this is for example this is the user ally is the last object that you saw on the map okay so this is basically a polygon with six points also you can have access to all the polygons so it's named user r i s don't get info so in this case we have uh uh two polygons right let me draw another one here so we have a total of three allies and if you run this one again exchange so now it's being changed to this one but if you want to get access to all of them you do use this one user allies and if you scroll down right usability three features like the first one the second one and the third one okay so this is how you can access to uh the draw the shapes that you do on the map and you might be able to use this to do filtering of os engine data catalog if you want so this is how some of the tools for the gmap next i'm going to show you here how we can convert google's engine javascript to python so for those of you if you're already familiar with the javascript and you want to try out python then you might need to convert your script from javascript to python actually right now the gmail python package has low things built in so it's actually very easy to do there are two ways you can convert the javascript you can either use some just assign whatever javascript here to a new variable and then wrap them with a switch double quotes so you can go to the google's engine website and you can copy any script for example you can just simply copy this one and then come back to here you just simply paste your content here it can be any script and just execute after that you can call the function gmap js snippet to python and you can pass in this variable uh at the new cell equal to that means we are going to generate a new uh code block with the uh python script and you can either want to import or you can just use those default and just hit one then you're supposed to see this one so this is the script that has been automatically converted from javascript to python then you can just uh shift enter to execute this one right so should see the similar result that you see from the javascript so this shows you a way to easily convert words engine javascript to python besides using this way uh if there's a an easier way that you can directly use the uh the toolbar so what you need to do you can create a new map right so once you have the new map then you can click the tool icon and then click this one okay so once you have this it's going to pop up this dialog then you can just copy and paste any uh python script from the earth engine documentation website for example you can copy this one or you can find any other one you like so for example for this one i can just simply copy this then come back to the notebook okay right click ctrl v to paste this and then just click convert okay it's going to convert this one into uh python script then just hit one you simply see the data being loaded here to the map right and then you can use the layers icon to turn the layer on and off or even change the layer opacity so this is how you can convert google's editing javascript to python okay so the first section just basically uh some introduce some gmat basics and and the next section is about google changing data sets because uh in order to do computation you need data set and you need to know how to access the google's engine data catalog it's very very important so next let's go through this example how we can load the data set so every data set in the google's engine internet catalog has a unique id uh for example this is for the usgs dm and so it's within the id is either single code or double quotes but it needs to be wrapped within the earth's engine either image or for example visual collection so this is basically a master data set uh just one single image sometimes you might have a stack of images that they could be called image collections so if you want to learn more uh you can go to the gross engine website to learn more about those but for this simple example we're just going to add the data let me just execute first uh you might be you'll be able to see the results right so we have three raster data layers earth engine image a dm link cover data set landsat and also vector dataset as a phrasal collection for the u.s states then you can set the visualization parameters so this is basically telling google's engine how you want to visualize the data set and for example this is for the elevation data set the minimum the maximum and also the palettes if you have used some uh uh desktop yes for example axis or qgs then you might be familiar with this uh basically you need to uh if you want to visualize the uh layer you can set up for example the range of the values minimum maximum and also what color you want to use to visualize after that you can just use map dope add layer so within here you can press a shift tab on your keyboard to bring up the function definition so again within the parenthesis shift tab and then from here you simply see the function definition what what parameter you need to provide and also some of the default values so this might be useful if you are not familiar with the function for example for this add layer function we have five parameters so the first one here is the object the second one is the visualization parameters and then the name of the layer and whether you want to sort layer by default or not and also the opacity so this is the one right the earth engine object the visualization parameters the name of the layer so or or not and also the opacity so we have this uh four data layers and this is how can visualize the data layers on the right here if you click this icon to turn the layer you can turn all the layer off and then just turn them on individually right so the second one here this is the dm you can change the layer opacity if you want from the left to the right and this one is the lane cover uh also the landsat data and last one here is the vector data set right so this is how you can load the data onto the map but uh in order to load the data you might need to know the id sometimes if you just uh if you're new to google's engine you might not know where oh okay how do i find these ids are you have no idea then here you need to search the earth engine data catalog right so you can either go to the google's any website and click the data set tab from here you should you can search any data set or you can scroll down for example to any section and then click and then you can navigate you continue to click in i click you should be defined here the unique id so this is an image collection uh basically it's a stack of images right let's think about uh on your computer you have a directory within your directory you have a lot of images right so this is just the name of the directory in gmap uh you can also easily uh search the os engine data catalog so what you need to do is to create an interactive map right and after that you can click the left here and then click data so from here you can search you can type any keyword to search the data set so for example a division right and then hit enter it's automatically go to the earth engine data catalog to search the data set and you should be able to see here the the list of all the data that has the education uh tag or you can simply select from any of these for example i can select srtm digital elevation once you select the data set from the drop down list then you should be to see for example the thumbnail also the earth engine snippet and if you want you can click the link here to go to the earth engine webpage to look for more information after that okay okay let me do this one more time education and then select from the list srtm you can just simply click the import button so if you click the import it's going to import these two lines uh into the notebook right so this is basically easy with it if you don't know the id it's ultimately going to create this one for you and then you can directly load the data onto the map you can also change the variable because by default all the variable will be dataset underscore and then three random characters and this is mine this might not be what you want uh you can change to any other name you like but make sure they also change the name here when you add the data layer so okay so uh since i already have this line that changed to dm i can just uh dd this one and then just shift enter to add this one to the map by default if you don't pass in the visualization parameters it will just gray scale so you just look like this and this is not very helpful because uh you cannot really see anything from this gray scale image so it'd be nice to you if you want to see um to have a visualization parameters you can you can customize a minimum maximum and then you can pass in the visualization parameters in this function so and then shift enter again usability this is much better compared to the default visualization besides searching the earth's engine data catalog you can also use the datasets module within the gmap package so all we need to do just to import the the dataset and then i'm going to shift enter once you uh create the import the data set module then you can use the dataset to find the data with auto completion so all you need to do is just data dot and then press tab eab on your keyboard it will bring up the list so here you can usually see the entire earth engine data catalog all the lists and you can choose any data set you like right so it's very easy to do for example if you here you can do maybe try for example usgs and just hit tab again right you can scroll down to find any data set you like for example any dm will give slow down [Music] is there any dm you like so maybe let's uh find which one geotogo topo 30 or this one right star tm similarly you can specify the visualization parameters and then you can add the data layer to the map and then shift enter very similarly you should have this one loaded to the map so this is how you can easily source the os engine data catalog and load data to the interactive map lastly uh you might want to use the inspector tool right so because all engine everything just from the os engine data catalog being streamed to your computer in real time so but you might want to for example query the data set uh you want to get the pixel values you want to get the polygon attribute this is how you can use the inspector tool so this is the first icon here in the toolbar and i'm going to create this notebook example this loads this sample data set here then click the icon click the inspector then you can click on the map anywhere on the map to query the earth's engine data set killing mine so this is when you click your mouse on the map it's going to get your mouse coordinates and then send the coordinate to google's engine to do the query and then return the results showing up here in this dialog box so if you click other location you will see here we have we have four layers right uh activation length cover landsat data and uh states you click outside there's no uh polygon data set so you only have the master you can also turn the layer off for example if you are not interested in for example u.s states or the land cover or even the landsat data users want the dm then you can turn the other data layer off and then if you click on the map it will only show you the the reasonable data layers you can also look at the coordinates for example the uh long steel and lactic also the pixel resolution right so this is when you assume you can also zoom out and zoom in the pixel resolution will be uh we will decrease and this is how you can use the inspector to query uh the earth's engine uh all the os engine dataset that you are loaded to the map okay so this is the inspector tool you can do it for any vector data set or master data set that you want okay so this is for the section three uh os engine data set next one i'm gonna show you here how to do some visualization okay so you can depends on uh the application requirement uh gmap here provide a couple of tuesday for you to visualize the data set interactively the first one here we're going to introduce is the protein tool right and it's the second icon in the toolbar again i'm going to load some sample data set uh so this one here we're going to use landsat data and also hyperion hyperspectral data if you once you the map is loaded you can click here to take a look at the data layer right so the landsat global scale and also the hyperion then you can click the protein tool right so from here you can select what data layer you want to do the protein if you for the first one here is landsat and then you just click your mouse on the map it should create a interactive plot right so we have this is the spectral signature we have a five we have six uh spectral bands in here and from b1 all the way to b7 if you click for example water you you should get different spectral scenes where you can inspect any location you like this is for hyper multispectral data you can also change to the hyperion high hyperspectral data and then similarly you can click on the map you can so you better see that this kind of spectroscope so this one is way more detailed it has almost uh 200 bands okay so you can click the map here again this is just for two sample data set that we're using here but you can use this one to create profile for any raster data set you want next one you can also change the layer opacity i i think i already showed you this one earlier it's very easy to do when you can load the data set then you can click the layers icon then you can turn this one a slide to the left to the right to change the layer of pasty you can either also change the base map layer so all the layers right now load into the map and they are all kind of interactive widgets so you can change them interactively if you want also you can visualize master and vector data easily for example let's take a look at this one here and if you click all right so this one right now we have two data layers the lanes data and also the dm right what you need to do is to go to this one click the layers and then click this icon to you can customize for example this is a uh rgb uh three bands right you can also just throw just one band and then if you click this one from the drop down list you can select what kind of band you want you just hit apply it will just show you one spectral band so if you want to change to a different band combination for example i can change to b5 b4 and b3 hit apply then you should see this one you can also change the opacity or you can change the range you can also change the gamma uh if you want this everything can be customized so this gives you a quick way to visualize visualize master dataset without even to write any line of our code you can also click the import it's going to automatically create the uh visualization parameter so that you can utilize this in your source code so that you don't need to use the interface uh if you don't want to so you can just click close so this is for the multispectral uh data you can also do it for a single band in this case we are using the the dm and you can click this one here by default is going to use the visualization parameters that you already provide for the data layer but you are welcome to customize so in here you can select for example how many classes you want uh if you don't select by default is some some of the classes might be 255 you can also on the right here you can choose the color map so you can select any any color map you like and let me try here i'm going to use the terrain it's going to change and then just hit apply it should be changed you can also change the palette because this one here if you scroll to the right you have a lot of patterns right sometimes you might not need like many you can choose for example 12 you apply are you very much similar you can change for example to other data layers uh i can change to maybe try this one yes hit apply all right and you can also change the gamma change the opacity everything can be easily uh customized so this is how you can um visualize dataset using the interactive gui the next one i'm going to show you how to visualize the vector data so you just execute this is the vector data the state of polygon for the us and you can similarly click the layers icon and then click this one so here it should be to be able to customize the vector symbology for example by default it's using the black color outline and also fill the color with uh some kind of transparency you can change here for example the outline color maybe to red color and you can change the opacity to zero it's going to show you just the outline if you want you can swing the opacity you can change the fill color maybe to blue color and just hit apply that's it very simple and easy to do you can also uh style by x view for example you might want to create a map that style the actually based on what certain xp values you can click this one and it's going to show you for example the columns available in the attribute table i can select for example name and then from here you can select for example this just shows you the uh the column values you just need to easily just applied very simple to do you can change for example to uh reason right this one has five values you might want to have five classes and then you can do you succeed apply you can change the opacity by different color different reason here in the us if you want to add a laser you can also add a laser you can just click this one and then you can change this for example class one to uh you can you can you can change the the text and after that just hit apply you will see here is this one is being added so it will show you for example class one this one if you change the text this for example north east and then mid waist midway or south something like that so this is how you can easily customize the vector data data set you can also for example turn the layer on and off during my the laser is tied to the layer so if you turn the layer off the laser will also be turned off and that one is using the interface but there's also function edge style vector so if you have a vector data set uh you can also provide the visualization parameters and also the pattern so in that case you don't have to use the interface so this is just make you it's more uh precise that how you want to visualize the data set okay so you use this way uh you will see this one is being loaded to the map so this is how you can customize the symbology to whatever way you want you can use this using the style function what column you want to use and how how you want the color to be used next i'm going to show you how to add the laser right you can uh there are lots of lasers that you can add to the map uh sometimes if you you if your map is a symmetric map that you want the user to know like what it's called represent then you might want to add a laser right so google's engine or gmap has some built-in legends if you want to know what kind of building available you can execute this code block to see some of those are common ones for example the naocd and also some modis data set next here this is example how you can add for example the naocd dataset and also just use the add laser function and then just passing the building laser naocd you should be able to see the laser so this is very easy to do right split here and once you have the laser then you can use the inspector tool you can click on the map usability for example the uh the length cover classes for example 82 95 and you can scroll down to see for example 95 represent what classes click on the map this is very useful especially you are working on some language length cover data you might want to add the laser besides the built-in if you have your custom link cover data that is not the standard one like any ocd you can also add your custom lasers all you need to do is to create a dictionary and the dictionary has values and also this is for the label uh the label going to show here on the legend knows the color so the color is the one here on the left this rectangle showing you the color so i'm just going to execute and these two looks pretty much exactly the same this one use the building but this one used the uh customer legend that you can provide a color and also the label and you might want to use this one if your data set is not the standard length cover uh product besides acting a lesion right this is basically categorical data you can also add a caliber for continuous data especially for um elevation and uh service temperature something like that so first let's create um an uh t format and this is how you can add a color bar so color bar and you just need to pass in the visualization parameters so the visualization parameters uh you can have a minimum you can have a maximum you can also have a palette so those things we wanted to utilize to create something like this you could have this color bar here on the on the map right if you turn the layer off the cloud bar will also be turned off by default is using the is being added to the lower right corner and you also have a background but you are welcome to customize for example if you want your color bar to be vertical you just need to add the orientation parameters to vertical and if i can execute again you should be able to see this one at the lower right corner here you can also change the background to transparent so passing the transparent background equal to two control enter uh so this one it might be because my ipad leave it is not uh most up-to-date so the transparent might work as expected but if you update your ipad leave it to the latest version then you will support the transparent background similarly you can use the discrete right so this is the continuous color bar but you can use the discrete parameter to create discrete color bar just like this right so if you map indeed it has the discrete multiple categories then you can create something like this easily okay so these are some of the basic uh tools you can use to visualize data sets and so the next one here is some of the tools that you might use to create some split panel map especially if you want to compare some data set before and after some so uh some event for example nasal disaster or you can just use the function split map and then passing the left layer and right layer if you're just passing a string those can be something from the base map so if it is available then you can use the for example a setup basemap and the google map and this is very nice for you to compare the result besides those uh for example google map you can also use the naocd right nfc for example 2016 and 201 so this is how you can visualize the result if you zoom to some location they with uh dramatic changes for example this is nothing near last figures right so we see from 2001 to 2016 some of the urban expansion and also the uh water dynamics so this one here is just simply the base map layer but you can also load agous ng data set so it has two d layers but you need to convert uh us engine data set to a time layer so after that you can pass in those two layers into the map here as the parameter and visually look exactly the same but this one right now the data set come from the google's engine data catalog okay directly and this one here the dataset come from the usgs wrwn wmsr web service although they look very similar but behind the scene they are very different besides creating the uh split panel map sometimes you if you want to have more uh you might want to create a link maps so in this case we're going to create a 2 by 2 link maps and behind uh the diversity is using the uh centeno too but you can have different bank combinations for example natural color and laying water so this is how you can customize for example the visualization parameters this is a list within the list you can pass in how you want to visualize and all the maps are linked together so you can easily move if you move one of those it's going to move all the other maps so this is how you can quickly visualize the different combination of bands for certain images okay so besides are creating link maps you can also create time-lapse animations so if you have you want to see uh the dynamics of the landscape using the landsat data archive you can use it using gmap to create something like that so i can show you here so this is kind of the animation i created you can just play you simply see for example the uh animation so this one shows you uh you can click the toolbar and then you can source any location um let me maybe change to i quality and then you just need to draw a rectangle on the map and you can customize those up parameters easily and you can also change different band combination you can change this start here you can change the end year if you want you can also create some aspects of indices normalize different index to visualize the data layers so this is how you can create a animation uh like i said if you go to you create any maps so let me show you here you just press b it will create a new one and then and then just map equal to g map dot map okay so this is the icon on this one here the globe icon so if you click this one and then you should be to create time-lapse animation for any location around the globe next one is create time time series uh composite because uh sometimes it might be used to get access to the original google exchanging data set for the time lapse here the one to show you about the ones just for animation if you want to have access to the real data set then you can use this icon so the next icon here this one here you click this one this one is creating um time series in cloud free time series images for example using uh modis or using sentinel or usda and name imagery uh it's very easy to do if you want you can try it out here i also have a sword animation showing you how to use these two and you can click play honestly okay so this is here you click the uh the the toolbar icon and then you can select from the drop down list you can also simply draw a rectangle for example on the map and customize the bank combinations all you need to do i just select this from these options you can choose the year you can choose the end year and then just hit apply then you have access to the i'm serious image opposite you can use the time slider to play with the slider or you can just drag the slider to any year you want and this is just one easy way that you can create our free imagery without writing any line of our code for example uh you can also do that for a centenary imagery for the same location right so the sentinel imagery is 10 meter resolution and the laser is a 30 meter resolution so depending on your specific application requirement you might want to try out different uh sensors and also you can do it for one if you're in the us you can also have access to the one meter resolution name imagery again you can change the band combination easily and then to create i'm serious images for your area of interest okay so this is for creating a time series images and so for the section 4 we talked about how to visualize the dataset and how to change the data opacity how to add some simple map elements for example laser node color bar also how to create uh link maps where you can utilize the uh map for interactive mapping okay so the next section is about data analysis so besides doing visualization you can also use t-lite google's engine api to do uh big data analysis and first example is about descriptive statistics so for example if you are using the landsat data or whatever raster dataset sometimes it might be useful to look at the minimum maximum or the mean standard deviation of a specific band then you can use a gmap function called let me let me let's load the data set first this is the lane set 8 imagery you can locate for example the spec uh the properties we can utilize the property names and then target info to get what kind of a metadata of this uh imagery you and once you know for example what kind of attribute it has for example link cover then you can use the dot get to look at the link cover gmap has a function image props to give you a shortcut to get access to all the properties so you can just hit one and you should be to see all the value like all the all the properties and also the value of that property so for this specific image the cloud cover is 0.05 so that means uh five percent you can also do image stats so it's going to you can you also need to provide a scale so uh uh to me faster we use a smaller scale 90 and once you execute you should be able to get for example the maximum value of each band from b1 all the way to b7 right so the value of each individual band the maximum the mean the minimum and also the standard deviation also the sum so this might be useful if you want to take a look at the some descriptive statistics of the the data set next one is about zonal statistics so if you have a vector data set you want to calculate some of the basic information about its vector polygon with and you based on a raster for example this is uh we're using the the dm and we also have a polygon using here is the u.s state boundary right so if you want to calculate they say the minimum elevation of each state then all the average elevation of each state then you can use the zonal statistics function within gmap so you can set the output directory and you just need to call this function passing the image and also the vector and the way you want to output in this case we're trying to calculate the mean and also the scale so you can first pull for demo i change this to a large number but you can change this one to a smaller number and this is for example for the dm right 30 meter and it only takes a couple seconds to actually calculate and the csv is being downloaded to my computer and you can open this one to take a look at the csv uh or the mean value right so for all the states you can look at the mean elevation for each one this is just for one single band you can also do that for multi-spectral band right so for the lens data we have a multi-spectral band you can calculate for example the sum of all the pixel dm values within each state within each portion again we can open this one and usability have multiple bands right so we can get all the value easily using the zonal statistics function this one here is just for one one category because you just treat all the pixels within the polygon is the same but sometimes you might want to calculate zonal statistics by group by group means for example if you have a length cover data set you want to calculate the length cover composition think it's polygon then you need to use donor statistics by group in this case we are using the nlcd right and we think uh the vector here is the u.s state using the zonal statics by group you can find out for example within each state what's the percentage of uh urban land karma what's the percentage for every culture right or other land cover data type you can easily use this to quickly get some idea about the lane use line cover composition within your study area okay so the data has been downloaded and now we can open and take a look at nlcd csv it depends on how many classes you have in your data set right so class 81 and for different state you will see the total area or it can be percentage so depending on how you set type the function this one we use uh the sum but you can also use percentage if you want right so either it's sum or percentage and this is how you can do zonal statistics uh by group the next two example is about supervised clarification and unsupervised so uh this is kind of a long tutorial uh you can you can go to the gmat python package if you want to watch the video because uh we might not have time to get into the details of all the uh this classification here but i show you some of the basic steps that you need to do in order to do supervised or unsupervised classification right so these are some of the simple steps you can follow through and i can quickly for example load the data set to the map and then so assume that we are trying to classify this image right so once you load the imagery then you can create some training data set so the training data set you can specify a region only by default if you don't specify it's going to create training samples within the image polygon so we can create some random samples and if you click here see this one you should be see all the sample the points right so these are all the random points that goes engine created and then you can use those random points to create the training data right so next you need to initiate the cluster so in here we use the k-means right so we're gonna for example in this image we're going to create five clusters and then we're going to train basically to extract the pixel values for those random points and after that you can just uh class the uh classify the image and then add the result to the map that's it so once you have the map this is the unsupervised classification result you can assign the color you can label a label the cluster because those clusters are random so if you want to assign a fixed color to each one you can so what you need to do is to have a laser color and the key and after that you can add the laser to the map right so once you execute this one you can change the color this is just a simple demo so the column might not be ideal but you can change the for example the name of the cluster is vegetation urban or water something like that and this is how you can customize uh easily you can also change the layer opacity right so like in here if you want you can let me turn up the layer right and shrink this to basically overlay the two layers to see uh the final result but you can also if you want you can create an interactive widget just like this one here you can also change the layer opacity using the ipi widget it's the same as this one lastly you can export the results so in this case i'm going to export you can use the ee export image to export the result to your local computer and then you can set the scale so length is 30 meter i choose 990 meter because it will take mass all the time to export the result of or coarser resolution data set okay so this is for unsupervised clarification and the other one is supervised so this is so-called classifier earlier is the cluster and this step is pretty much very very similar so this goes through example again we're going to use the same data set so this is the landsat 8 after that you can draw a polygon to create 20 samples if you want but you can also if you already have some existing training sample you can do that in this example we're going to use the naocd uh land cover data set so we can add that nlcd to the map and then we can generate some random training samples uh in this case we're going to do 5000 training samples uh might not be displayed on the map so you can turn this one on and basically we join the random samples and then we're going to extract the label right because for supervised clarification we need to have the label and we use the naocd as the uh reference dataset to extract the label and you also extract the value for the original uh landsat spectral band then we can use the training samples to sd to change the classifier and then can classify the image right so again change the classifier and lastly classify the image that's it oh the steps are pretty much similar we don't have time to go into the details but you can follow the example here you can also look at the google's engineer documentation to my more details once you have the image then you can set the color right so if you want the color to be similar to the one of the nlcd data set uh you can so you just need to set the properties to the image and after that you can add the data set so right now this is the one this data set is the supervised clarification that we created you can compare this one to the maybe you want to maximize you can compare this one to the original nlcd right so this is the classified length cover look at this one it's not perfect this is just a demo but you can have a better training data set to have a better a length cover clarification but it gets the work done uh easy using google's ending just a couple of seconds then you can get the classification result and again you can visualize the result you can also export the result to your computer if needed next i'm going to show you here you can also create training samples earlier this one is based on random sample but if you want to manually create a labels you can do that as well so you can use this toolbar icon here to create i also have a quick demo here show you how to create training samples using gmap right so if you click the map and then you click the hang icon then you can easily create for example labels you can change to different color for example these are all water and then you can change for example to vegetation then you change the color you draw rectangle or polygons on the map to create the training samples later you will be able to use this training sample to extract pixel values and then to change the classifier if you want this one is still a work in progress so i haven't made a tutorial about this yet but this is a quick demo showing you how you can create training samples and then you can export the samples uh to your computer and then you can open those using other uh desktop software if you want right so in this case you can export the data as a swipe file and then um open using qgs okay so this about data analysis and this is just some simple example there are no more things you can do using google's engine uh more more information can be fine on the google's engineer documentation webpage there are a lot more things you can do uh there okay so uh next i'm gonna show you how to create some maps right so if once you've done all the analysis and visualize the data you might want to create some maps and you'll be very useful to create some high quality maps that you can use for your zone publication or technical report so in this case we're going to use the carto ee module so the calorie module is uh was developed by our co-host ko markus so and he uh integrates the mode into the gmap and now you can directly use gmap to create a maps uh easily so let me just quickly show you the uh example so cartel pi is no uh install yet and you might need to install this one on your computer so if you want i can copy and then so you need to basically open the anaconda prompt and then just conda activate on the activate and then to geo type this one and then ctrl enter so this one sometimes might be very challenging to install on windows computer because i developed this example using my linux machine i haven't tested this on windows so i'm not quite sure if it will be installed or not if it doesn't then you might need to try um a different computer it looks like it's okay so this uh if it works okay so now the uh the color pie package has been successfully installed and what we need to do is to actually restart the kernel restart and clear all outputs then you might need to go to the import the libraries again and then navigate to here to this section and let's import this one okay it works and just execute execute all right so this is how you can create some maps nice maps using gmap and you can also have the color bar here to the right and this is for at the global scale but you can also create a port rgb imagery for example uh using the laser data right so those data sets come from the google's engine data set so let me execute and we can take a look here okay so you can also add a coordinate grid to map right so you can have the labels you can have a nose arrow and you can also have a scale bar so and this is the package right now is still being developed right now it's very basic but in the future we might have more complicated [Music] scale bar besides eating this you can also add for example a laser right i can show you here this is a little bit more complicated but you can just follow the example and then customize your data set and then change those parameters to get the map you want right so in this case we added a laser into the map you can have a coastline you can have a city on the map you have multiple data layers if you want so this is how you can quickly create a map like this lastly you can also create a plot multiple maps right so if you have multiple data layers you have a raster layer you have vector data layers you can combine them all together to create assumption okay style let me see here okay so for this one here the data set uh you need to have the the sample data set from the gmap repository so you can either download the entire repository from the gmap for example from here and you can clone the repository to your computer the dataset is basically under the examples folder and data set so for me i already downloaded this one but because the dataset is not under the same directory you can just copy the entire directory and then come back to here then let me see here if i paste my data set uh here i and so within my data folder i have the countries on straight file and i might need to actually modify this one just to the data set uh directory and then you can hit one you should work okay and then just uh figure so in this example we have the uh this uh data set but we also have the uh country polygons if you want to create a map that with multiple data layers then you can follow this example right so you click this one this is the simple example you can also change the projection so uh if you want some custom pro uh projection uh you can do that as well right so this is how you can create for example using uh different projection this one is uh equal uh rules and there are a couple more projections you can choose from uh you can let me quickly go through the example here this is using the service temperature data and you can use the different visualization you can customize the bounding box and create map using different projections so i have couple examples in here you can for example to customize prime messages to create a map for the projection that you like so this is the first one second one this time my santa might take a little bit longer because uh basically you are trying to request the data set from google's engine and then to do the reprojection so it might take a while okay so two more examples and this is the low vision and so anyway you can customize the data set to any data that you like and then to create maps like this you can also change the location of the um the color bar what else you can also create time lapse animations so this one might take a little bit longer to run because you can download multiple uh time series the data set so essentially you're trying to create time-lapse animation but uh you can have a coordinate grid you can have a scale bar or other stuff added to the map so if you want you can execute this one and let me just quickly show you what the results are look like and it's going to download all the images from the quotes engine data catalog and to composite this one as an animation but this one we have 17 years but you can customize this one to uh earlier here the end year right uh just to save time so i just changed to 2000 but you can change to 2020 if you want and after that you should be able to be an animation like this easily and it's going to download all the imagery and then composite them all together okay so now the uh time lapse animation has been created successfully so we can go to uh the downloads folder and then click time lapse you should see this one here the gf animation you can double click and then supposed to see this one the animation right you can have a title you can have the uh noise arrow scale bar and also the coordinate grid right so this is nice if you want to create some animation like this so they can publish on social media or insert into your website okay so that's about uh creating a maps and then uh next section is about export data uh you can export an image or you can export on image collection you can also uh export pixels so let me just quickly go through here there are a couple ways you can export the data you can export the data directly to your computer or you can export data to google drive so if your data set is a small data set you can export data to your local computer so in this case we have to lancer data right global scale but you you can simply draw a rectangle somewhere and then this is the roi right so you can basically the user roi and then you can set the output directory for example the downloads folder and then set the output file path lastly you can just clip the imagery it's going to simply download the imagery to your computer as one file okay so just the lancet then you can open that one using uh qgso or arcgis you can also if this is a multi-spectral band you can export it each band is one single five so five per main equal to two that means you want to export uh multiple bands it's explained as one image okay so now you have lane say for example b1 b2 all the way to b7 if you want you can control place to export this one to your google drive and then you need to log into your google's ending account to look at the status of this one you can also export for example the entire image collection right so if you have a time series images you might want to download those images all at once so in this case we have seven images right so you can just use the gmap export image collection it's going to download those images automatically onto your computer if you're using the javascript it's very inconvenient because you need to click the button one by one you need to click the button seven times in order to download the imagery but you'll be using the gmap you see everything is automated so uh saving you a lot of time you can also export pixels as a numpy array so if you want you can simply define an roi and then directly export those as numpy aware then you can use those numpy red for follow analysis you can also actuate for example a pixel values based on a swap file or points right so in this case we have the landsat data for example and then if you want to export um the pixel values of all the u.s cities you can just simply you can download the sample data set and then you can change the straight file convert the strip up to us any object and then you can use the object to actually to extract pixel values uh if you want so the sheet one on the dm hello to the u.s cities em sweep file oh i forgot to execute this a call block so if i execute this one again and then we should be able to run this one now so basically we're trying to extract the pixel values of all the elevation of all u.s cities okay so now you can turn this one uh you can open the uh the strip file on your local computer you can also export as a csv if you want okay so we don't have time so you can actually execute this one later if you want you can also export a freestyle collection so basically a vector data and this is how you can export the data set from what's engine you can export the vector data to your local computer or you can export to google drive so you can just use e to shp to export the data to your computer you can also export vector to google drive so there are a couple formats you can export csv or can also export as json or or kml and last section is about developing web apps and so we don't have time to get into the details this is if there are two ways you can develop web ads you can also use uh ngrok so this one you can use the local computer as the the server to also web apps you can also deploy to heroku it has a free version so you can also utilize so we have uh two video tutorials that you can watch to follow through if you want to develop our web apps i think each one is like 15 20 minutes so you can uh follow through the uh the video okay so uh that's all for this uh workshop i hope you will learn something new and uh useful and uh so if you have any questions you can post the question on the github repository gmap and then discussions i'm happy to answer your questions okay thank you so much for our attending and hope to see you in the video videos take care bye

Channel: Qiusheng Wu

Views: 13,945

Rating: 4.968504 out of 5

Keywords: earth engine, geemap, geospatial, gis, google earth engine, ipyleaflet, ipywidgets, jupyter notebook, landsat, mapping, python, remote sensing, tutorials, jupyter, dataviz, gee, satellite, google earth engine tutorials, google earth engine python tutorials, geemap tutorials, python tutorials, ipyleaflet tutorials, folium, folium tutorials, geospatial data science, data science, programming, dem, earth engine tutorials, GeoPython 2021, GeoPython Conference 2021

Id: wGjpjh9IQ5I

Length: 91min 2sec (5462 seconds)

Published: Fri Apr 23 2021