Introduction to the Earth Engine Python API

okay so what I'm going to do now is talk to you a little bit about how one produce an other way of interacting with the world the way we are trying to do this so far has been through the core editor and that's not the only way in which one can work without engine and doing so in a way that is based on the Python API gives us opportunities that I think are very interesting so rather than start with something which is about configuration and setup and skills and so on I will start by trying to show you a set of visualizations I thought you did try to do using the Python API so here's this is a doable within Earth engine in the core editor but here's a point I want to make I think the thing is extraordinarily capable of doing really hard stuff sometimes it's frustratingly difficult to get it to do the simple stuff so for instance you can get an image you can have a time series through which you want to do an animation but you want to put the date that's not easy to do in attention at all it's frustratingly difficult and for you to take every frame out and find some other way of annotating those images and so on to make that more unique useful and so on is not easy whereas if you were doing this with the Python API it would be really simple to do this and rather than just try to peel it with you that that's simple like I'll show you a few examples so I've been trying to see how I can visualize something like the module here is a rich source of data this data on satellite images that show cloud cover the this daily data there's data on decadal time series at the de resolution on precipitation there's data sets on meteorological dark drought where evapotranspiration is greater than precipitation so they they're all there at a daily scale and what we are unable to do is visualize them in a way that I think most people are interested right now like Cape Town has had in the past in India there's a serious drought affecting drinking water in in some of the big cities in India and large areas are being affected so ordinary people are really interested to know what's going on what what are the patterns of monsoon how does the drought how's the summer progressing and these are things that remarkably a map allows you to see and you really don't have to be an expert to see this so I've been trying to just visualize and tell stories with data so there is a whole set of analytics going on behind the scene but at the end of the day it's a video that tells you a story about a pattern and so this has been possible to do for me but lastly using the Python API so here is an illustration of what one could do with the Python API I'll come back into the setting up and all of those things in a little while so so what I started to do is this this this is data from this year there's a University of Tokyo data set that just tries to visualize drought meteorological drought so there's there is the hydrological drought and there is the ecological table so this is just one dimension where you know the these are a series of if evapotranspiration exceeds precipitation play starts to get drier and drier and that's what this is visualizing and there's a daily data set that's tracking all the other parameters and trying to build this so this was again something that shows how as the summer progresses and and if there is a rain that the conditions are changed as you see towards June where we are the monsoon starts to come in the south and that that's where things start to change so this is something that's already very interesting for people to actually see how an entire at a continental scale how a place dries up and and and starts to get wet again so but the reason that's happening that's beginning to change is that you have this annual pattern of the monsoons and there are large datasets that actually can help you construct the monsoon but when I went and looked I couldn't find something that showed me what is the monsoon where did the rain start how much rain falls in different places and when does it fall all of this actually can be seen it's data but you can actually see it and when I put it together actually thought the monsoon was something that was only about India but not true it's something that affects a large part of Southeast Asia as well so it's it's it's more the south and the southeast asian monsoon so this is the chirps data so look at what happens in June July so interestingly this is called the southwest monsoon the one that comes in June now but if you look at the way it is coming here the front that of rain goes from the southeast to the northwest see again the that's the trajectory so where people got very engaged and started asking but this is it's the southwest monsoon why is it coming in from the southeast so that is an interesting question and I said that's that's something that's worth looking at but in any case the point is when something as large as this is affecting such a vast area periodically the entire area is pulsing to the timing of this and here it was again a very interesting opportunity to try and bring together a dataset put together by citizen scientists this a large publicly accessible data record of when the pied cuckoo which actually is also which spends part of the year part of that population is in southern in Africa and in East Africa and they come across the Indian Ocean people think that they come on the monsoon winds into the Indian subcontinent and then they start moving ahead of the monsoon so we have a 1500 year old legend that talks about this bird being the bringer of the monsoon and there is this whole mythology about this bird never drinking from the ground this is a bird that's supposed to drink directly from the sky so when this bird gets thirsty the mythology says the legend says that this bird starts to call him which is also the time its coincides with the breeding time when it starts to vocalize people say this is when it starts calling for the rain and it won't drink from the ground so the clouds had to come and let go of the water and that's when the cuckoo quenches its thirst so people in northern India so there's a resident population in the south so in April the resident population is found in the south so my animation starts in April and goes up to the end of July the monsoon starts around June so as you go so good at in April there are no birds in northern India so as the migrants start to come the monsoon winds start to grow from across the Indian Ocean from Africa those are the winds that are supposed to be what the cuckoos track and come into the Indian subcontinent and then they continue to move further up not they nobody knows whether it's the resident or the migrant population that moves north but the kuku start to move to northern India right now in April they're only in southern India and they start to move so here was the opportunity for me to put two very different datasets one is a remote sense data set on the timing and onset of rain the other is a data set on the distribution of reporting dates the first reporting dates for a species so when you put together what I found was actually very interesting you can see for yourself so the cuckoo does go a little ahead of the monsoon and people start seeing and hearing the bird and that is and this is a very very old observation so here is something that connects feet on the ground which is people collecting the ordinary people collecting data on on reporting birds there are the eyes in the sky and then you have an entire mythology that connects people through this so this was a story that helped put together very very different kinds of datasets so now coming back to the issue of why was this called the southwest monsoon because there's nothing that seems to be happening from from the West so it turned out that people said it's probably the wind and that's something that was not trivially easy to animate and visualize but that was an interesting challenge to see where the wind comes the very turn how what is the wind direction across the entire subcontinent over the over the year so this was again something that I was able to do see what happens to the winds towards the end of March and to the towards the end of May and June so the ticker the lines the faster the winds that's the monsoon so a lot of the question of why it's called the southwest monsoon gets answered there and then you have the northeast monsoon it completely changes direction and starts moving back in the opposite direction from from after October and those are the winds that on which species are the migrant species like the cuckoo are supposed to go back and their dragonflies there are many many species that do that Raptors it's it's a really interesting pattern so more simply here is something that is very trivial you have a series of images that show you a true color image of the entire Indian subcontinent so there are clouds that are there or someplace or the other every day so can you see the monsoon as it comes if you were to take a view from 500 kilometers up there so this is the view you get of the entire monsoon so these are all real images of of how the Indian subcontinent looks or the entire year as the monsoon starts to come so this is can you see the greening behind so this precipitation and things are changing so what is happening is you're seeing everything from above the clouds but the action the change if you are an ecologist that you care about is happening beneath the clouds could you look underneath the clouds this was the question I was asking because this is where the land is completely getting it transformed so this is where with the earth engine you can actually do that and the next one tries to show you that so you can just using the median reducer you can do this so that's that's all I did so here is how so I was able to take a whole stack of images for every day across multiple years and say on a given day this is how the this FCC the false-color composite looks let's see if there are bursts of green that coincide we know that the rains come in in June so let's see what happens so I mean there's a lot to look at at this map in this map you can look at the the the lakes altitude lakes of the Tibetan Plateau you know freezing and thawing and this is so much happening here but what I generally want you to look for is a browning of the entire subcontinent towards the end of April and then with the with the coming of the rains but the bursts of green that you see at a huge scale that follow the precipitation that comes to the monsoon okay I'll stop here so this was my attempt to sell you on why you must care about the Python API because it has all the analytical power that Earth engine provides and something immense flexibility that a language like Python allows you to do so for you to get the I'll leave this on for a moment you can get the link and to the to the presentation and B to the slide deck at the link below so when we're talking about using the Python API API stands for application programming interface it's the way you interact with a platform like Earth engine so there are two questions and I think the first one is why I mean if I'm having a hard enough time trying to get our heads around the code editor in the browser why another thing the other is how so I'm trying I'll try and touch upon both of these and at the end I'll also leave you with a couple of things to start you off if you want to use this to get the Python API started and for those who've been interested if you really want to bring what earth engine can do with R and you're really interested in making the to talk to each other Python API is your friend anyway you can do that right now is through the Python API and it's not hard at all as you will see ok so why use the Python API so this is something you've already seen this is the entire Earth engine platform with its geospatial datasets the algorithms and the storage and computing capabilities all of which is in the cloud so the only way you interact with this is that you send out requests and you get back responses and results so this is the way you interact with this so what is happening when you so this is now a familiar thing already so many many things that that mediate the way you interact with this platform is already built into the core editor there is a window in which you visualize your results if you would like to look at them as text there's a console if you want to see them as maps there's a map window there are places to hold your scripts and do the versioning there is a documentation there are assets that you may want to bring in at various points there is an inspector there are so many things that are built into this already but all of this in some sense is is and the way you are doing this is is through JavaScript so there's already this is your the code editor is giving you a JavaScript API to earth engine so that's basically what you're doing so the browser is basically your window to connect to the earth engine platform in the cloud and what you're getting there is a JavaScript API now you could just take that out and replace that with a Python API that's essentially what you are doing but once you do that things start getting more complicated so the the code editor is the same for all of us troubleshooting is really easy we get the same thing if I share my code with you get to see exactly the like David showed yesterday we have to see exactly the same code exactly the same interface everything is the same so it's it's me it's it's wonderful but there are situations where you may want to trade off some of that uniformity for some customizability and and other kinds of capabilities so with the python api there is a one code editor kind of interface that you have in the browser you can have multiple windows through which you look at Earth engine so for instance you can have something this is for those of you are familiar with Jupiter notebooks you could be working with Earth engine you know Jupiter notebook kind of interface so that's one interface that uses the Python API and allows you to interact that engine using a Jupiter notebook interface this is something that Google has started in the last couple of years it's called collab notebooks these are notebooks but these are notebooks in the cloud again they are like your book they're like all of you are familiar with Google Docs you have your spreadsheets and your documents and so on this is a jupiter notebook in the cloud it just sits like one of the files in your Google Drive and you can open it you can start working you get but it's a Python environment it gives you a Python environment and it really really takes a lot of the friction out of of creating and installing and maintaining Python which is which I will come to in a bit which is really the hard part of trying to get a Python environment going and this should be the most familiar one for many of you who use our studio so you could be doing a lot of this in fact nearly all of it within a studio so I I do some of my stuff where I write my entire thing in corporate and run stuff especially when I'm trying to do something with the outputs that I can only do in our alright I do better in our that's where I'm doing this so I'll show you how this this can be done so just to get a sense of the trade-offs so this is not like Java Script is better or python is worse it-it's a it's it's an entire landscape of trade-offs and I think for specific things that you may want to do specific kinds of of costs and benefits in these trade-offs might make sense to you so the great thing about the JavaScript API is that it's simple its integrated it's powerful it's very easy to share scripts but then if someone wanted to upload a certain kind of you have a shape file and you have you have some obscure format from which you want to upload into Earth engine it's it's there's a lot of friction you have to then do you'll have to do a series of conversions and and you know added to SX and it's not a simple thing to do whereas in Python you probably could write a couple of lines of code and get kml of whatever format in which your data is into Earth engine very very easily it's with almost no friction and if your data is continuously changing let's say you're getting points from for which you need to be sending alerts it was there a fire here or not you have a stream of data points coming in then it's it's much harder for you to integrate something like that with Earth engine you know in a way that can can I to rate whenever there's a change and update to your list of data points and there's a new coordinate coming in you want something to run this is not something you can do in straightaway in the JavaScript API so if you have different kinds of inputs and outputs that you would like out of your interaction with the Earth engine and you want to be able to customize the kind of things you want to do it's very limited what you can do with the JavaScript API so the Python API is is as powerful in fact it's it will be found yeah if if you are already some familiarity with the JavaScript API the Python API looks nearly identical if there isn't very much more of learning one needs to do to start working with the Python API but it has a huge range of customisation options it is still easy to share scripts you can that's not hard at all and the vast range of input and output options but then the really big cost is that you have to have a working Python environment that you maintain on your computer and this for people who have Macs and Linux and windows and so on it's all different and everyone's Python in one is different and to maintain that Python environment in a way that can work for you is not is not trivial and I think that is the very big cost that one has to see in trying to put all of this together so now having dealt with the white Python API I will leave you with a couple of examples of how to get started using the Python API using - two kinds of options to interact with the earth a to interact with the earth engine the first is this the Google collab notebook so this is a link that works if you have a Google account in which you have signed in and you have this link this should open for you you should be then able you should then be able to save this as a collab notebook within your Google Drive so that should be simple so you save a copy within your own Google Drive before trying to run it and make sure you're logged in with the same email the Google address that you've used to register with Earth engine because there is a part of authentication that has to happen which will require your login and all of that so that that would just make make this a lot simpler so okay so if you have got if you have kicked through and this is something you can follow along this is it's not too difficult but if you want to pay attention you will have the notebook with you you can do it whenever you like so it's so those of you how many of you are familiar with the notebook metaphor first so notebooks are so those of you who have worked in are have written code but people have told you that it's useful it's really important to document code so to say not just what you're writing but also why you're doing something like this because you know you even if you're not collaborating with someone else who has to read through your code you're going to be collaborating with yourself later so when you go back to your code six months later often it's very hard to find out why you did something this way because you've learned something else and you have another way of thinking about it so it's useful to document so the the iter ative process by which we do our science requires us to be able to make certain decisions document why we are doing things and make the entire thing actionable so it wouldn't it be very nice if you could write say the the entire logic of your method section to a paper with the code that was actually generating the results that follow so if all the filters are in one document where that would be wouldn't that be very nice so that's exactly what the notebook paradigm the metaphor tries to do so you can have chunks of text which you can format and you know like like this so this is a text cell so basically it has cells and each cell you can run code within each cell yesterday someone was asking about being able to handle a code you know tangent line by line it's it's not really lying by line you can do it cell by cell and that's so an object created in one cell is available to actions that you're planning to do in subsequent selves so it is it's it's very very helpful but the key thing here is that there isn't a place where you can look at your map easily and there isn't one way in which you can do it and since this is largely a geospatial analytical platform that's a very important thing we have to be able to see what we are trying to do and and with that so the way it works is you will need to get a few you get a virtual Python environment that that begins to run so I don't have to do anything you can connect to that virtual environment but you don't have to specifically click on this so I maybe I should make this yeah a little is that final would you like me to make it bigger yeah okay so okay so here are you can run each of these code cells so there are text cells which you can just read you can double click on this and you can edit but that's not what we are trying to do so here is a code cell and you have a little play button which you can use to run the code code that is there so right now you're getting a Python environment without Earth engine the API there are things in API so what you're doing here is you're setting up the environment to include you're installing the Earth engine API and another library called folium which is used to visualize the outputs and another small helper library which I have so it when you click on this you'll start to see that it starts doing things and this should work for all of you if you're following along you can pop it on your live in the beginning oh oh let me so the link to the collab notebook is on the last slide of this day can I take it off everyone done okay yeah so there is a warning and an error message which I'm going to ignore now so what it has done is that it has installed the earth engine API and now the thing that happens quite seamlessly when you use your code editor is that you have logged in and it asks you to Authority authenticate and all of that but right now this is not it doesn't know who you are and what you're trying how you're trying to access earth engine whether you are so whether you are an authenticated user so you need to do the authentication and this is the way it's done here so it gives you a familiar link that you will see in a moment and here is mine so I have to allow that and then I get a token of some sort which I then have to put into the next cell I have to paste it and all of you will get something that's different and then I can authenticate it what happens is that it then stores that token and for this session in this node it's only for the session you have to do it again the next time you come back to the world book if you've shut it down it saves the authorization token and now I'm ready to start using your engine so I have got the library that I need I have done the authentication I'm ready to go so I am just going to I need to initialize this which is done using this EE dot initialize and here is something that will print which will tell you if you have managed to initialize successfully or not you need to have gone through the steps above to get to this successfully so now I would urge you to look at this a little more at your leisure don't don't maybe you should continue as we are going forward so all I'm trying to do here is I wanted to visualize a map so this shows you how using a folium map one can visualize some data so this is from a national park close to where I live where there was this pretty big fire so I have both a two color image and a shortwave infrared false-color composite that shows you the area that got burnt earlier this time so this is this gives you this gives you a an interface and map interface what I have done here is I have added two layers it's it's a little different from the way you would do this in JavaScript but I think it's it's not it's not too hard to catch on and there are you can the way you get information about most objects in you can you can find out using the type command in Python what what I have created a sentinel image here i have loaded a sentinel image here or sorry so that's the s2 so in python you don't need to declare variables with the VAR keyword so it's telling you that this is an earth engine image and now if i wanted to find out how many bands that does this image have there are ways in which i can just get the lengths so there is this dictionary that you get when you do get info and this can you do this with great discretion because it can be this this can end up being very large objects so in this case it's getting the s2 image getting the info about it pulling the list of banks and counting the lengths and reporting that it's sixteen bands in that image so now to do something less trivial so what would be an example of something you're trying to do where you are really not trying to sit in earth and do lots of stuff all you want is you want to look at a relationship between two things let's say you wanted to look at the relationship between rainfall and maximum Evi for a for a large area so you just want to see if I mean yeah ecologically what I'm giving is a trivial example but still if you're trying to do something like that what you would like is the last data set that Earth engine has and you know that there's a jobs data set which gives you precipitation you want it for the whole year you want some way of aggregating it say the total rainfall so you sum the whole thing 365 days of charge data that is sum and then get for each particular parcel let's say you have 50 different places for which you want to get the total annual rainfall over aggregated over the entire area so what you want to do is get the total yet one image reduce it over a polygon and get the median let's say the median rainfall within that polygon median total rainfall in that polygon and then you want to just put it in to correlate that against Evi values for which let's say you have a whole year stack of avi data from modus and you want to look at this let's say with the maximum Evi and then get a median or the maximum maximum Evi for your polygon of interest so that's what I'm trying to do here so what I have done is I've got the districts of South Africa so for every single district I am trying to get the I am trying to get the sum of the rainfall which gives me a book so that's for the whole year so this gives me annual rainfall for 2018 and for the same 2018 period I am trying to get the maximum Evi so what I'm going to get is for each district name I'm going to get total average rainfall annual rainfall in that district and the max the median Evi median maximum PPI for that district for the earth aspect so I just couldn't I'm going to get a data frame if you're in the RF kind of thing with three columns so that's what I want I really don't want to go into Earth engine and generate all of this and then worry okay I'm going to export this CSV and then bring that into I want to visualize this maybe you can do all of the similar engine you know itself and you're fine with it but maybe you really want for your thesis you really want to put this graph and you want to generate in GG block or you want you're working in Python you want to use something else like say out there which is a visualization package so I'm doing it's almost exactly the same kinds of things I'm creating an image with two bands now I'm using image cat I'm using a reduced regions I'm using the districts that I have uploaded so this is already an asset I am using those districts and I am getting for each district I'm getting total rainfall it says median but it should be Mac CVI anyway so and I'm going to put that all into a data frame and when I run this cell yes the output I get that's it so it's processed an immense amount of information and given me just what I care about without I don't have to do any export and it's it's already an object available in memory now which I can work with as I would anywhere else and in the next step I just have to call that object and plot a graph so it's it's it's really very simple to do this so this is great to go lab notebooks which are notebooks in the cloud so now something that's very similar but works in our I have almost the same so I have another link to a notebook share so you can download that know about our notebook and and work with that as well if you would like but that is not going to work like core lab notebooks because you'll have to set your entire you'll have to setup your Python environment you will have to do a lot more backing work before you can get the our stuff going it's not terribly difficult but it may not work it's very likely that you go and work for any of you right now but it's there once you get your environment set up it's very simple so the part that you need to do is to set up your environment so there is this I'm using the leaflet library in r2 to visualize reticulate allows Python and our to talk to each other very very easily and then I'm using the tidy verse from where I want to use ggplot so just to show that I can actually visualize something from Earth engine right inside our studio oh yeah okay yeah so this is using e-flat I'm just looking at the night lights and image it on night lights and it's interactive I can do all of those I can zoom in all the things that you would expect to be able to do can add multiple layers you can do all of the get the same kind of interactivity that your and Glenn had done something even more fancier with wrapping this into a shiny app and so on this is this is for simple iterative kind of work that one would want to do and I have exactly the same code that wasn't the collab notebook that's here so I run run it virtually it's identical the way it runs and this is the output that I get but now I'm going to plot that in ggplot and that's really simple as well so you're without having to go to any of these and these are these are really simple way this is more realistic about how an ecologist who's familiar with the are and so on is likely to the reasons why they're likely to be interested in a data set that is so rich a computing platform that has so much capability to run so many in both of these it's reducing like satellite images from each day of the year for a very large for the entire country of South Africa and giving you summary statistics by district which you can then plot really very easily so I hope this is somewhat useful in terms of how we can try to fit this into your workflows and if there are any questions or comments I'm happy to take them but that was basically what I wanted to show you so yeah I'm sorry no that's fine that's not shared with you but I shall do that yeah so what what you're seeing is a visualization of the data it's not the data itself the bands that actually went and you really can't get it from our Python it's an RGB visualization of those bands okay any any other questions or comments but if you'd like to take a little bit of time and work and now it should it should work there that error should have gone away I've shared thee I able to get to work now okay otherwise I think we are right on time for oh there's one more session of course the dance of lo okay so this is this is a requirement if you want to get to the next temple tensorflow you mean the the way this is going to work is that you need to be able to do it in collab notebooks so it's good that you now have an idea of what a collab Norbu cos and how to get earth engine going in that I think I'll hand over to David from there

Channel: Alanna Rebelo

Views: 10,198

Rating: 4.9805827 out of 5

Keywords: Google Earth Engine, Python API, Monsoon, Rain Bird

Id: IGUYSeoeHhg

Length: 44min 39sec (2679 seconds)

Published: Wed Jun 26 2019