Introduction to Google Earth Engine (End-to-End Google Earth Engine)

hello and welcome to this lecture on introduction to google earth engine google earth engine is a cloud-based software platform for remote sensing in this talk i'll tell you a little bit about why this platform was built how is it different and what it can do so let's get started here's the outline for the talk today i'll start with a brief history of google earth engine and why it was built and what problem is trying to solve then we'll look at some of the early applications where it was used to solve the problem of clouds and then i'll dive into demos if you've seen any of my lectures i like to keep my lecture very interactive and show you stuff instead of just talking about it so i'll do some live demos and show you how google earth can be used through a variety of platforms and then at the end we'll look at some of the case studies and applications to give you a flavor of what kind of things are possible using this platform so if you've done remote sensing before you are quite aware of this workflow of remote sensing where a bulk of the time is spent on preparing data downloading processing reprojecting and getting your data ready for the analysis and only a small part is used for real doing analysis and data science and this is the problem that google earth engine is trying to solve if you look at the data prep part this is something that can be standardized all of this can be done once and the data could be made available to the users so they don't have to all spend time doing things like atmospheric corrections uh converting file formats and so on and if this can be taken care of then people can really spend more time doing data science and this is the kind of basic premise of what google earth is trying to make it easy for you before we dive in further a brief note on the name a lot of you might have heard of this product called google earth and this is a very popular product by google it's been around for a long time and this is a really cool 3d globe where you are able to explore the world in this rich 3d environment it's got access to high resolution imagery 3d buildings and so on you can type in an address and fly around with it many of you would have used this product this is a primarily a consumer product aimed at everyone to just learn and explore about our planet why while google earth engine is a newer product it is inspired by google earth but it is primarily built for researchers and scientists it's a cloud-based analysis platform that has got access to remote sensing data the scientific data sets uh and it can be used to process and analyze this data and we are talking about google earth engine today so let's see how google attention came about this slide is showing the screenshots from google earth and when google earth was launched they had a feature where you could use the historic imagery feature to go back in time and explore any area and this is showing the amazon rainforest and this images are taken about 10 years apart and showing that a pristine rainforest has been gradually deforested and the people who were living this region they were really concerned and they said this is really important for us to track and they asked google uh can google build some tool for us that can help them monitor such areas more regularly and you know quantify this data well google earth is a great platform for looking at the high resolution imagery acquiring high resolution imagery and a regular interval is not possible and also it's a tool that is not useful for analysis so what google wanted to do was to kind of tackle this use case of monitoring and quantifying of the earth observation data and they set out to build this platform so how do you build this platform the first thing you need is data and there's actually a lot of data available this is just the nasa satellites that's been up in space and capturing data about us and there are many many satellites that have been capturing data and making them available to the public so there is data out there that can help you monitor the earth but if you try to use this data before you know it's really hard to access and even harder to analyze so the design of google attention was inspired by this book and this code sums up the the gist of the book quite well that when you're trying to implement a big data system like this it is far easier to ask your question where the data is and not the other way around if you think about the traditional software that are used for remote sensing that runs on your desktop and if you need to do any analysis all the data needs to be on your computer and that is much harder when you want to download terabytes of data so instead this book says you build a system where you are asking the question where the data is and the data can be in the cloud and all the computation can happen in the cloud and it sets out the vision for how the system like this could be implemented and that's what google earth engine did so the first step was to just get all the data in a centralized place so google earth engine now contains all of the archive as well as the current data and not only landsat but all of the satellites and the data sets that were publicly available they went and ingested this and all of this are available to you just by specifying it so there's no more delay you can just use any of these datasets readily and not only that all of these datasets are also updated on a regular basis so as soon as the satellite captures a new image and it is made available google has engine goes and fetches it so now all of this has been done automatically and there are over 40 petabytes of data that is in google earth engine and as a user of the platform you can just say i want to use this data set and you can start using it without downloading it processing it etc so great now you have access to all this amazing data sets is this enough to really do large-scale computation not really because to process this data you need large amount of competition and so the next step is to have a lot of computation power and the key here is that the competition and the data has to be co-located that means when you're trying to process the data you don't want to go to the public internet download this data and then do the computation the data and the computation are co-located and that allows for really fast data processing and you don't need just one computer you need many many more computers and that's what google did they put massive amount of computation and the data together and this is a photo of the data center showing you know one of the google data centers and you can imagine that this one of such data centers would have the earth engine servers and the data all of them at the same place in a recent talk google also revealed that uh earth engine it uses about 300 million cpu hours per year now imagine the kind of computation that is required that is supporting the earth engine user community so now you have the data and you have all of this computation power can you go and start building all these applications not yet because to really effectively use this kind of computation power you need to write code that is you know can use all of these computers in parallel and that is really hard so what you need is a way to specify your operations and say i want to do this operation and then the system should be able to figure out what is the most optimal way to harness this massive computation resources and that's what google earth engine api does api stands for application programming interface and api is available in many different languages and platforms you can write that instructions in javascript python or through third-party platforms like qgis and r and we'll see the demos of all of that so now you can understand what google earth engine is it is a collection of remote sensing data along with massive amounts of computational resources a powerful api that allows you to leverage this data and this computation resources in an effective way and then set of tools on top of that that can allow you to code and specify those instructions easily so a combination of all of this is your google engine platform so let's look at a problem that google used this platform to solve the problem they chose to title first was this problem of clouds uh what you're looking at here is the landsat 8 image over the andaman nicobar islands in india and this is the most cloud-free scene that satellite could capture over the course of the year now if you're trying to study this area that's really big problem because you can't see what that area looks like believe the clouds but the good part is that the satellite takes the picture of the same place every 16 days and the next time the satellite comes overhead and takes the picture the clouds would have moved the shadows would have moved and you know over the course of the year it takes many many such images so if you are able to combine these images and say for every place i would look at the whole stack of images and take the median value of the pixel so the high values would be the the bright values with the clouds the dark would be the cloud shadows in a median value at each pixel would be likely to be a cloud free pixel so if you do this kind of analysis at every place you will end up with this thing called a cloud free composite and now you have this ability to see through the clouds and this compositing technique is not new it's been used for a long time but doing such requires a lot of data processing and that's where the earth engine comes in where you are now able to do such computations for entire years worth of data for the whole world and that's what google used this platform to do and it created this the first ever high resolution cloud free image of the world now this image is great you can see all the places in the world but it's not a real image it's a composite image that's been created by selecting the median value of each pixel so there is no snow there is no cloud anywhere it's spring everywhere so it's not a real image but it's a very useful image where you are able to see how each pixel on earth looked on average through the year and now since google had this platform they said we have the data for last 40 years and can we just create such composites for all of the 40 years and that's what they did and that created this product called google earth engine timelines this is a really interesting data and i encourage you all to go and check out you can go to any place on earth and see how that has changed over the last 40 years i'll show you some examples this is the region of india where there is a lot of conflict going on and you can see the settlers are moving in gardens setting camps and over this large span of time there's been a massive amount of deforestation you couldn't really look at this kind of change in a short time duration but now you have this 40 years of consistent resolution time series that is available and you are able to look at this change that has happened similarly you can track urbanization this is uh the development of a new city in near new delhi and you can see over this large uh time span you can see the diverse changing force the the farmland being converted to an urban area the highway is being built and again this is really useful for studying urban growth urban change [Music] this is showing a famous or infamous lake in bangalore this lake is uh you know has a lot of pollutants and it often catches fire and you can see the reason why you can see the the development of the city and you know this massive uh road has been built on the lake bed and this has caused a lot of environmental issues around this you could also track large infrastructure project and how they'll change the the landscape this is showing the construction of terridam the point here is that this kind of product couldn't have been created using any other platform because to do this you first need a lot of data so this computation use about a petabyte of data even if you could find a place to store the spread effect of data on a hard drive and then you get the most powerful computer you could find in on a desktop and run that it would still take hundreds of years to do this kind of computation but using a system like google earth engine you are able to harness thousands of computers in parallel and you can then do this competition in a very short time and the exciting part is that as a user of earth engine you now have access to computational resources like this where you now can run this global scale analysis using a lot of data and that's really exciting so let's see how google attention works so as a user of earth engine you are sitting on your computer you are and you send a request to the earth engine system saying that you want to do this computation and that computing gets to the data center it gets distributed to all of these computers and each computer does a small part of the computation and once all the computation is done all the results are then combined and sent back to you so maybe you asked to compute a statistic and uh google election use a lot of data to compute that statistic but all you got back was just that number so you didn't have to download any data you didn't have to use any of your computation resources locally all of that happens in the cloud and that's what makes the system really fast and effective we'll uh this will all become very clear when we see the demos so let's just dive into some of the demos so here we are this is the google earth engine code editor when you sign up for google earth engine and you get your account this is where you end up and this is one of the ways to access the google earth engine platform here what you're seeing in the middle is where you write your code and this is how you specify what competition you want to do once you specify a competition you can click this run button and that request gets sent to the earth engine backend and then the earth engine processes data and sends you the results back and in this case this computation is saying that let's just load one landsat scene and that's what is displayed here so this is a map very all the results would get displayed on the right hand side this is the console where if you ask to print a chart or a number this is where it would get printed and on the left hand side is where you can save your work and access your past work etc so let's see this uh computation in this script uh it's trying to access this one mindset scene and you can see just you know loaded here without much delay but you know that's not you know a big deal you know you could easily load a single landsat scene any any other software as well uh let's see how we can do some computation and let's say we want to compute the ndvi we learned about the ndvi earlier it's a ratio of two bands the near infrared and the red band and earth engine api has a function to allow you to compute this band ratios so we computed this ndvi and say we want to load this layer so you can say i want to load this layer so now the earth engine servers computed the ratio of the bands from this image and displayed this image here that was pretty fast but again not very impressive but now what we can do is we can say i don't want just one scene i want every scene of landsat and i want to do a composite for the entire world and what that can be done because earth engine can now distribute this competition over large amount of machines and can be just as fast as it was doing on a single scene so let's see how this works so now let's ask customer to create this your composite from one year worth of landsat scenes and you can see as i asked for the computation the the styles are flowing in each style is being computed by a different machine so it's only doing that computation and it's just sending me the result back and you can see now i'm processing data for at a regional at the country level in an interactive way and similarly i can say don't just give me the images give me an uh compute the ndvi and give me that results back so now with just a few lines of code i am able to not only process this data but compute ndpi index and display that and people who have done remote sensing this is a game changer right if you had to do this in any other platform you would just spend months and months downloading and processing this data versus in just you know under 30 lines of code you are now able to do this at a global level and you know this is a really game changer let's look at another demo and this shows you how earth engine can be used to really work with this deep time series data here's an example that is using some precipitation data and what we want to do is compute the average rainfall over the last 30 years and compute the deviation of the rainfall from one year over the last 30 year average and again this requires a lot of data processing because we have to get all the data from last 30 years process it and then compute the deviation of the rainfall from the current year over the past 30 year average but in earth engine you can write the code you can run this and what happens is earth engine will process all this data and it will generate this chart and you can see this chart got generated which is for the bangalore region what is the deviation in percentage of rainfall or in during 2017 year which was a recorded rainfall year versus the past 30 years and you can see this just took a few seconds and this is the power of earth engine where you were able to do this really complex long-term time series computations in very very fast and again you can display such data on the map as well this is uh the code editor where you are writing javascript code and getting the results maybe you say i am a python programmer i i want to program in python so google attention also gives you access to the python version of the api i'm now in a jupyter notebook where i can type python code and once i install my earth engine api library i'm ready to access that in a jupyter notebook so here i'm doing the same computation where i am writing code to get some sentinel 2 data over a year doing the composite and computing the ndvi and you can see the code looks very very similar to the javascript one because the api is exactly the same whether it's in javascript or python just the semantics of how to write code that uses the apis changes slightly so either way once you learn the api you can use it through any platform now you can see this map here this is you know now i'm in python notebook and i'm accessing earth engine and i'm able to look at the same compute the ndbi using the same set of code you can access this earth engine through a variety of other platform as well one of my favorite ways to access this is using qgs so here i'm in qgis and many of the gis and remote sensing users are more comfortable using this interface so if you are working with qgis you can run your earth engine code in qgs as well there's a plugin that's available it's called the google earth engine plugin once you install it your qgis python console is now able to run google as engine code and the advantage here is that it can read the data layers that are loaded into qgis you can combine the qgis python api along with earth engine api and do lots of interesting things where you can create maps or create animations using the qgis api and earth engine api so let's just see how we can run the same examples that i showed you earlier into qgs so you can see now this is the qgis python console and uh here we've written some python code that is using the google earth engine api and as i run this code the again this gets converted to a request that goes to the earth engine back end there's in the back end computes the tiles that we need and the dials flow into qgis and you can see now a new layer called image is loaded you can see the tiles are being fetched from earth engine and what this is is a one state in india and we are asking to compute the composite for the entire year and now we can run the code again and say give me an ndvi for this entire state and imagine the kind of competition that you need to do an ndvi for the entire state but this is now using the the geometry of the layer in qgis sending it to earth engine it's getting ndvi computed for that geometry and the results are flowing back and now we have a layer representing the ndpi for the state we are in qgi so we can now use the qgss print layouts create a nice map and you know we are done instead of downloading processing all this data we use google earth engine to do all the heavy lifting and we got the results of qgs so coming back to the presentation let's summarize what google attention can do google attention can work with collection of images both temporal and spatial collections of images it can apply filters you can select the subset of images for the region you want for the time period you want you can apply many different kinds of computation on it and once you apply this competition you can either create a map like we did and create an ndvi map or you can compute a chart or a statistics the way we did with the rainfall data so this is the kind of the workflow of earth engine where you are able to work with this large amounts of data and generate the results that you want so let's see what people have done with this and earth engine is around for a long time it's primarily used as a research platform research tool and if you just go to google scholar and search for earth engine you see there are over 9000 publications that reference this tool and again all of these are research that has been done in various domains and they've used google attention as a tool and i want to highlight some of the my favorite applications that show the diversity of the platform and the capabilities of that another powerful aspect of the google earth engine platform is that your users are able to create this rich interactive applications based on the data sets so not only you can analyze and get your analysis results but you can actually publish them and build this interactive web applications very easily and that allows you to really share your data in a much wider to a much wider audience and they can explore this data so i'm going to show you some of these applications and case studies and that can give you a flavor of what kind of things you can do with earth engine so let's just look at some of this applications the first one i want to show you is uh there's something called global water dynamics and this is a global analysis that was done using about 20 years of landsat data and what it was trying to look at is how surface water has changed over this 20 years and you could go to any place in the world and it you know this is powered by earth engine so all the results are computed in earth engine now you can explore this so you can click anywhere and you can see the color-coded classes where you can see what part of the area was permanent water what was seasonal water there are patches of here where there was water loss uh there are areas where there was water came and you can see here this is a reservoir in in karnataka in india and you can see that you know the levels in reservoir there is less water during the summer period and during the the monsoon months there is a high amount of water so you can get this kind of global level statistics using earth engine and users can really explore this data in this bridge details really powerful data set as well to study the the land surface water dynamics another really useful application is this something called global urban heat alien explorer if you are using remote sensing to study urban areas you are aware of this phenomena called urban heat island where due to the urbanization the area urban areas experience a much higher temperature and now this is an application that uses the modis data and it analyzes this data and tells you what is the the temperature difference between the urban areas and non-urban areas and this is the city of bangalore and you can see if you click it around anywhere it's generating this chart you can say this area is about you know 1.3 degrees higher temperature than the surrounding areas and now you can explore the history of this phenomena and also this the spatial as well as temporal aspect of this data so all of this is powered by earth engine where you know you are able to carry out this analysis and publish this app that users can explore another thing that is was built using the the radar data in earth engine was this application to explore flooding in 2018 the kerala state in india experienced massive floods and one of the users from the earth engineering community quickly put together this app using the before and after sentinel one images and to show where the flooding has happened in relation to the population so it allows you to select the district that you're interested in and it shows you the population data in the red here and then the blue part is the flooded india and this allows you the the decision makers to really see where to focus their effort pitch areas uh are more severely flooded etc and you can see this is a really powerful use case of earth engine where you are able to use this and process this data very quickly this is another really important scientific study that was done where they looked at the vegetation cover change in the tiger reserves of india and tiger reserves in india has the highest level of government protection and they spend a lot of resources so this researchers wanted to know where a certain area when it was declared tiger reserve does the vegetation cover in the diagram improve or decline and they use the long time series data from 84 to 2012 and build this app where you can now go and look at this area in detail and they built this very nice split panel app that can look at the data on the left side it says you know after declaring this area as a type reserve did the vegetation condition improve or decline and whether this hurt or helped the vegetation cover and you can see there's a this linked publication that came out of the study and also uh not only you know this is the useful part where you do this research uh your research doesn't just live in a journal paper but it can be a live interactive map where the decision makers can actually study and you know it can help you communicate your results in a much better way this was also one of the recent app that was built during the kobe times when the lockdown was imposed in india that was also coincident with the harvest time in majority of the northern india and the researchers were concerned that the farmers would be able to sell their harvest and they built this tool to help the government visualize and see when a particular area would have a harvest season and this is all using the modis data and you are able to do this analysis using google app engine so the next i want to show you this application called global forest watch remember when i showed you in the beginning that google's engine was inspired by the request for people to track deforestation and that's exactly what this app does where it is as soon as a new satellite image is captured it is able to analyze this and compare it with the past images and if it detects deforestation it is able to provide an alert and this is an operational system where users and people who are trying to monitor this forest they can go in and you know sign up for the area the notifications for the area of interest and as soon as a deforestation activity is detected in that region you get an alert and this is an example of a deforestation and illegal road that was being constructed in the forest in peru and you know the as the road was being built the the uh the system triggered an alert and they were able to stop this road construction if such a system was not there they would have probably known about this kind of illegal activity years down the line in february right so this is an active uh monitoring app that is powered by google earth engine google as engine is also integrated with google's ai platform that means now you can use google engine along with tensorflow and build state-of-the-art deep learning models the way it works is that you can use earth engine to sample large amounts of training data and then this training data can be used in the google's ai platform to build a deep learning model and once the model is built it you can run this model in earth engine and do this global scale prediction and this is a really powerful system which allows you to leverage the power of deep learning and apply that on satellite imagery i want to leave you with links to some useful resources i want to show them to you so you can know where you can find additional information if you're interested in learning more about earth engine the main resource for you to learn earth engine is the earth engine user guide the user guide contains a lot of information if you look at the guides tab this contains step-by-step guides along with code examples say you are interested in doing machine learning in earth engine you can look at the machine learning section and say i want to run some supervised classification and that means the guide will have some videos it will have the step-by-step instructions along with some code samples that you can copy paste and run and that's a really useful learning resource i also recommend looking at the tutorials here there are a whole bunch of text and video tutorials that can help you get started and these are all really good material that is suitable for beginners we also have this thing called earth engine data catalog and this is the repository of all the data that is in earth engine while you can access this data in the code editor the earthing catalog itself contains a lot of useful information about the data set so maybe you don't know what is the right data set for you to use so you can come use this uh data catalog to really learn more about different data sets and you know you can browse through this here learn about different data sets and once you uh find a data set that you are interested in you can each be each of these data sets will have a information about the different bands that are contained and at the bottom you will also have some code samples that means you can quickly copy paste this code into a code editor and look at this data and that's a really useful way to get started with some of the new data set that you may not be aware of the last resource i want to show is this repository called awesome ge this is a curated list of learning resources related to earth engine and contains everything from the blog posts and the tutorials to different resources and repositories that can help you learn earth engine so among these three resources i showed you you have plenty of material that you can use and get started with art engine that's it that's the end of the lecture hope you've got a good overview of what google as engine is and what it can do and hope you are all excited to start learning and programming in google attention thank you

Channel: Spatial Thoughts

Views: 4,814

Keywords:

Id: kpfncBHZBto

Length: 32min 54sec (1974 seconds)

Published: Sun Nov 29 2020