my question you are not on here but not on local that's going to be a problem isn't it why local yeah over our logo which is great I will function the other way let me join STR to injure time after time you know where I keep that on macro news so just swapping to bro can you convince yourself again well beers it up now great okay yeah come on Google engineer and I think your vocation power good hi I'm enabling summit I do not yeah all right in and on okay um I'm should not meet me I lost a couple more so none of you have taken on one-on-one before correct oh okay I'm going and one of the founders of Earth engine that's Pierre he's our project manager that's Kurt he's a back-end engineer and that's Jeff he's also back in the ship you can all have your own personal so the condensed version of this is a little different in that it less typing more concept and starts off with a conceptual overview my thinking is that the more you know about how we were thinking we built it and more you'll be able to intuit how it works and have to ask less questions that's the theory so jumping into our overview if you didn't already know we're a big data processing system we are inherently parallel that means that you get to use locking up the machines on our side and you don't have to manage them we do all the management for you so you typically don't even know you're using parallel system other than it's a little bit your shape but we'll run your job over hundreds of thousands of machines and until you achieve is also a fast it's designed very specifically for scientists and not suffering tears so long a software engineer and I've been doing remote sensing for thirty years and I have worked that entire time with scientists and I have a pretty good intuition about what they can and can't do or more importantly about what they will and they won't fit so we designed it for from the from the point of view of what we design just what to do is to staff and how do I want to do it and approach most things from that direction and what that means is our main goal is try and make them easy I'm trying to use easy as possible to use large-scale tools again without having to do any machine management we try hard to make it so that you know defeating Ariel in some cases we just lose you not we don't get it perfect you upload assets in so you can do a little bit asset management because you got to put in somewhere and you got to be able to call them something but for most of it we try hard to make sure you don't have to do a lot of management tools let me name it stuff basically we try to remove all the IT problems and in addition to scientists we try to enable non-traditional easier so people that probably can't write a lot of code we may try to make it easy for them to for the scientists to build tools for the muse and then cases for them to access the stuff directly without having to write any code at all my team specifically is trying to focus on society's biggest challenges things like import station drop disaster disease but you can use a platform print anyone this is just where we spend our efforts trying to make a very large scale all right changes on effects on very large scale problems we don't believe that we know anything about deforestation but we know people that know a lot about deforestation and we just want to remove this loss that they have so I know how to do IT I built a bunch of tools for doing IT and you guys can use them and not have to do the IT suck okay one of the things that it makes it really useful is that we have a ton of data so it's all processed and ready to use we have every Landsat image ever taken we've got all of the motors all central one all simple two and a bunch of other satellite data sets as well we also have a large collection of new physical data forecast data climate models both forecast fan hyn casts and sister generally a bunch of data and it's all data that people have asked for so so what shows up and says you know how many ocean color data I can't use you well that's that's the wrong way to go about it you show up me say I want you submerge in color today you go get it for me and we will fire equipment machine go download it processed it in there and then they're ready to use and hopefully a few days and now you've got it to use with all the other data sets and didn't have to do the downloading and file formats and conversion and projection molesta so one of our big selling points is our public data catalog when you use us you've got a couple options we have a JavaScript API and a pipeline API the JavaScript API is attached to our interactive code development environment so our playground code editor and people have successfully used the Jaypee JavaScript API we've known as well there's a star there because it's not really easy if they have made it work but you know you can get there we also have a Python API which is just a Python module and download install and people build web apps using App Engine this is essentially how they're doing it they're go guiding to Python that means this cloud hosting environment people use it because it's generally free when they should a huge amount of traffic and mostly these applications don't get a huge amount of traffic so it's just a very easy place to host your web apps and then people have been starting to be able to use it with Jupiter notebooks which is an AI Python notebook system this one has a star too because when Tyler someone pushing on this and when Tyler started he said I got to work with giver notebooks it's 37 steps to get there and I'm or anything talk to me when it's down to 9 or less you see 210 right now and so he's trying really hard to shave at the rest of it so that journalists are not a huge endeavor to get set up so all that said what probably went through your brain is free JavaScript I'm doing Python why would I bother dosage and the answer is that it is very powerful and very very empowering to be able to do an interactive code development so type in a little bit of code click run to the results I can workout click run sees results and the interactive environment that we've got makes that one set turnaround on a lot of code that you're writing with the Python API you have to make that work yourself did you for notebook will make a easier but as I said that's couple steps and more importantly it's trivial to get help using the JavaScript API where it's a little more work to be able to send someone and your notebook and them to open it up and get in there and do they have all the right modules installed it can be more problematic so we always recommend that we start with the interactive code editor just because it's the one with the most support it's got inline Docs it's got inspectors it's got fastener active legalization and the translation from that to Python is trivial we intentionally make the two API as close as possible so that we can just switch from one to the other it's worth noting that if you're hoping to use the Python API with things like pill and numpy type I it doesn't work that way we were insanely tensorflow is tensorflow notice which is you build a computational description and you send the description onto the server and the server doesn't know anything about pill and imply or senpai it's running all its own code going back in so it's not a triviality to just use those two together with our arbitrary Python modules we are trying to make it easy now to bring data into numpy but if you're working with terabyte data sets you really can't bring a terabyte into numpy and expect it to work so there is always going to be some limitations about what you can do in your Python client which is where elating modules are learning this is our code editor looks like that but we're going to spend the rest of the second half of this we'll talk inside of that learning how it works all right so I want to introduce you to some big concepts about how we work and what the data model are so that it will be easy to understand the rest of the talk so our primary data model we process data how we data model is how we're able to abstract away a lot of that and our primary data model is a feature and what this looks like is it is exactly a geo Jason feature if you know what that is but you don't know what that is this is essentially a row in a table with a bunch of properties and one of the columns in that table happens to be a geometry so you can sticker printing hot lines and polygons into that column and then anything else you want to the other columns so lift the properties and the geometry just like a table runner table just like one element of a shape file just like one geo Jason feature they're all the same thing this is a map from one of the tables we have this is major conservancy ecoregions each one of those polygons is one row through this table and they have different properties in this opposite color on properties by coloring being what type of by almond thing so we get a feel for that the next data model is that of an image images are also features in that they have properties they also have geometries but they are special as they have bands if you've ever worked as a Landsat image Landsat images come with 899 bands at 30 meters and one band at 100 meters so thermal bands of a different resolution than the other ones we have the same model bands don't have to have the same projection they don't have to have the same resolution but they do we do in fact that they are all Co registered so if I drill through a stack of fans I expect them to all give me the same point on the planet maybe it's given me a one liter pixel and a 300 meter pixel on 20 kilometer excel but I still expect the result to be from the same location images also have in addition to a geometry a date because we expect to be able to filter and sort them based on dates it's a real common operation so two properties are required is the geometry and the date but then you can have all the other properties and creature in half and all the metadata we get when we download an image goes into there as well so solar adds an angle collection timer and gain settings all that stuff is available as columns in the table of images the final data model is a collection which would just bag to these other things so if you've got a bunch of features and you put them together into for instance a table that the collections collection features you've got a bunch of images and on your machine you would think if you put a bunch of images into a directory that's a collection of images we do this because our collections have semantic meaning to them we give you tools for doing filtering and sorting and joining on collections so once you've collected a bunch of things together you can do all these other things to it including two things called mapping reduce and these are sort of the key point to that where Earth engine starts to be unlike other tools so mapping is when you do something to each of a bunch of objects so if you have a directory of images and you wanted to compute the cloud cover of each of those images you might know how to do that you pick up an image you can see the cloud cover you write the pipe like out of another image could be the cloud cover like the bytes out until you do this for each image it has an important property that they are each independent so you and a herd of your friends could all grab a different image compute cloud cover and write them out together at the same time and that's essentially what we do for you once you are mapping over autumns we know that we can parallel ice app for you so you can map over 10 items or 100 items or million items and that gives us the ability to do a 10-way or a hundred day or amazingly paralyzation anyway we want to be able do it so examples of things that you might do with this is if you've got a bunch of features a bunch of geometries and when a compute the area of each one those are all independent operations we can operate them you just map away the roads in the table compute the area and stick the area backing to the table so this is a pretty typical kind of mapping of friction I just talked about cloud cover where you do be able to the same thing got hundreds of images and compute cloud for each one you tell us how to compute the cloud cover and then we'll go and do that for you over of collection of images and then you can map everything that you might not expect to be mapping over so if you want to collect creating a composite of images for each month so look at all the images for some place collect them together by mode and then squish them together into twelve composites you can do that by mapping over moon so you make a list of the number t1 through 12 and map over that the function would take a number take number sixes evaluate all the images for jube collect them and return image and because you're using the mapping construct we know we can do that in parallel maybe you don't need over 12 images maybe you're doing it over the entire stack of Landsat which is 32 years times 12 months and so you've got 390 items in your list of updates or something like that you can operational and with no computer all same time okay the converse of map is reduced so map is doing something to each element reduces doing something to all the elements together so if you have computed all of the areas of a bunch of geometries and you wanted to sum them all together that's a reduction they're reducing all of those areas into one sub area I view this is a bag operation you get some numbers from somewhere and there's a lots of places you can get them from stick them into a bag and then you know sort of put this back to a meat grinder you know the summing lead grinder take everything into the bag and computer sum or take everything this back and compute variance or take everything this bag and compute the Pearson's correlation typically there's statistical operations that you're doing in those cases but there are things that are not statistical that you can do through the same system I already mentioned the example of doing something very over with your features you take a bunch of images and you squish them down so that all the images from the same rug get pushed into one image you've got one image for the month at the composite and that's again a reduction per pixel reduction through the entire stack the system that we use to do all these reductions is called our aggregation system because you're aggregating stuff and we use it for other things that you might not think of as a reduction like training a classifier if you're building random forests you train the reinforce you of a bunch of training notes going out and collecting all the training points and shoving them into the classifier is an aggregation for us so all the rules that are applying for any other reduction apply for this kind of aggregation salt um we have a bunch of different ways to reduce and a bunch of ways to reduce bunch of things to reduce with this is just the slide thing we've got a bunch your walk through several of them were to come here so a real common thing is to reduce through the bands of an image to find min the max or the variance or something like that so at each point for each band get off the value those values go into the bag the bag it's process and one value coming up the other end so this particular handle will some where we tell me nice to gathering and get one value met the internet you can do the same thing over multiple images so if you've got three and it is in there three bands apiece and maybe they overlap a bit maybe they align and maybe they don't align in terms of extent you can drill through all those images and whatever numbers you happen to be intersect go into the bag and you process the bag and when I conduct event this is a really common operation and we know what you probably need to do in most image collection reductions is a curve in per pixel reduction so if you want the minimum of this image collection we will get all the red bands and finds the minimum of Detroit band at each point and with all the blue bands open minimum at each point from over the base so right or evil in that we rule out ten bands in 10mins out a thousand and a thousand is out it's a perfect so programmed kind of operation you can reduce neighborhoods where you say go get all the pixels around this pixel stone in the bag do some statistical analysis this is again per man per pixel you can reduce on a region so you say here's a park go get the mean and DDI for that park that's a real common kind thing it has also done per band unless unless you tell us otherwise and what I want to show here is that there are five bands in and they are numbered b1 b5 and what comes out the other end is also named the major remain what conductor in is main values so for each band that goes in and you get a value out on the other side and the names match we've talked about dictionaries yet but we will admit reduce regions is when you've got a bunch of these look at the table of a bunch of geometries then for each geometry go compute some image statistic for each one and what this potentially does is it autumn into your row with new columns so if I get the me and a little Japan and I've got my rows and I'm just going to mean column end into each row in my table so it's table in table out in this case because I put through an image ticket to get those things ask questions so give them maybe shape files on finality so that may be a multi-part shapefile polygon multi-part shape file and give me a multi polygon or yeah so do you have one record per polygon or can you conglomerate them so multi-part polygons the geometry standard we use understands multiple geometries okay so you can have a multi-point geometry or a multi polygon polygon so that's like this work and holds just as you expect you know I've got our geometry to hold essentially turn polygons got me I didn't understand that okay and we have ways to Union I don't not so good ways to dissolve for instance but you can Union things together to make multi-armed trees that's treated thanks reduces vectors is a real common thing people think they want to do so we've got some set a native image and you want to turn it into a bunch of vectors will make that easy but typically what people are then going to do giving it a call receptors are limited in learning some reduction on X vector that's we can cut that will stuff out you can actually reduce on the region's out here including the vector so as we figure out where the polygon is you can say I just want a mean throw the polygon way don't have a bottom make them that just give me the mean value and we'll do that and this lets you avoid making billion row tables that are just all of these polygons if you don't with polygons so pretty pretty common bet now this is image to table kaveri community's table but maybe you didn't want charities at all maybe the numbers so table can be just numbers so other stuff that we do thinking we know what you need to do if there's a lot engines like that a lot we think we have a community most of the time because there's observation stuff tends to work the same way most of the time so when you're doing van math you're adding to Landsat images or normal lighting to enhance religiousness with that we know that you probably mean to do things on a per pixel per band basis so take the two red bands to each pixel add them together into the groove and it takes a lot of typing together so so imagine in each image so we bend out from the output image we also know that if you are doing math with a one band image and a seven band image what you probably mean is you want that one band image to actually be replicated up seven times so if for instance you have a whole Lancer image and you want to divide by two or black I just spoiled my punchlines but we can write / - you want every single pixel in every single band you me / - and so we do this automatic replication for you all the way up from a single number or a single band this is called promotion in the system when you lost the promotion for me because we think we know what you want too so we'll promote a number to an image and then we look at it and say well you've still got seven bands over here in one bed over here so put another seven that image and then do our pixel by pixel band now talk about good questions yet okay so an example of some computation that you might want to do if we start with something like the lame-duck collection this is for Landsat 8 this is about a hundred thousand images you might want to filter that down so you get some easy right tool you build something such that if you had a big directory than a hundred thousand images you could find the ones that are between 2001 and the end of 2003 and oops copy you know convert the 800,000 images down to you know 50,000 images so that might take those 50,000 images and one can compute surface reflectance for each one so right now to do that you pick up each image compute surface reflectance and write out our new image in the surface reflectance of each image so that's a math operation obviously then might take the surface reflectance and compute the NDVI for each one of those and pick up each a middle image out of this pile of 80,000 service references computing the eye right out and into the eye image so you'd end up for many thousand service reference images to 8,000 into the eye images you might then want to composite all those NDVI images together into some maximum composite and so you pick up all these images and make one big giant image and write it out you might then take that big giant image and compute an RGB visual relation on it so it will sync greens and reds and Randy vitamins and you might take the whole thing and we project it to Mercator and get your answer out the other end if you're doing this in a traditional environment that's probably how you do it we don't do anything like this whatsoever we actually work from the outside in so we start you've explained or I've explained a step of processes where each step consumes the output of the previous step and this efficient build a computational description we work by taking a computational description and working backwards so we figure out what the output is you want notation wants a Mercator now and the the outermost layer is the function that converts to Mercator that gets sent to a server and the server says why need some inputs I need not expecting significant I'm going to be projection give me my inputs and there to go off to another set of servers and that server says well I'm doing an igt visualization I'm expecting one band give me my one band I'll compute it I can watch another server that server says well I'm expecting to composite together bunch of into the I images so I've been writing the I images that goes to that neck severing all the way down the bottom to get some actual data and down at the bottom of this the very first piece of the some particular graph says I got some data I can send what you've asked for and it passes up a piece of data and the Rapidan is expecting that processes does this little thought the Senate passes upon mentioned is able to process impact and run and nowhere along the way do we compute any of these intermediates we didn't storing images but in store anything on disk this is all an on-the-fly process and not only is it on the fly done title by Kyle we'll see that in a second but this ability for each of these processes to ask for input maybe it means a lot of inputs get this ability to scatter and gather so we scatter out to get all the inputs one at the end we're probably talking to a bunch of machines each of those machines can do a little bit of work and then we gather all the results back together enabling fast processing okay all right so hold on to that for a minute and we'll talk about Thailand when we hit a Landsat image the very first thing we do is cut it up into pieces for every image these are just tuned to 56 by 286 pixel tiles in the original images projection and resolution and band and bit that's all that we just covered the pieces so easy to work with small pieces we also build a down sample version of this so for every four tiles we average them together by weekly combine them by averaging into a Down sample tile and we do this over and over until we've got entire pyramid of tiles and so most of the time you're doing computation in Earth engine you're actually working somewhere in this pyramid unless you've explicitly said I want this to happen at 30 meters most of the time you'll be somewhere the higher in to discriminate depending on what you're looking at so we adjust what we're computing by what you've asked for and you implicitly ask for things when looking at a map and you zoom and pan around so as you zoom in we look further to put your down on this curve in but you can always just say I want all this to happen to 30 meters don't do any down sampling at all so we're doing a complication we do the same thing when you're looking at it in your browser it's on a javascript map the JavaScript map is also tiled so each one of these tiles get asked for independently and I give this in this case an automatic 16 way priority so in addition to the other spreading out what we've done over multiple machines we get additional spreading out here so we're really only computing one of these pixels one of these tiles at a time we're going to compute that tile we actually have a legend metadata where we can tell that for the that we need to work with those are the tiles that we intersect with we say any one computer down at the very bottom of that pyramid change by that computation trade only needs to get a little bit of data so doesn't mean to get all the little images just needs to get those four tiles of whatever land image is working out so we get a little bit of data we spread it over a lot of machines we do big scatter gather and in a few seconds usually we can give you the answer you're looking for as long as you're willing to accept down sample result who you're not then we get something else all right so the basics of how we work computational description we work on piles the river bunch of machines will get your results very fast okay so that's that now I'm going to show you we're walking through some examples in the code editor so you can see how it actually works in practice how's the questions you bet you know so what's the default down sampling if you see if you don't specify resolution I mean how does it determine that comes from the math what resolution do we use or yeah it comes from from what you are always asking for resolution even if you don't think you are so if you're looking at things on JavaScript nap then the map this map has a resolution and add the projection based on inputs or something based on output so the output is what to find it in this case I'm looking at a map Mercator projection that probably of a thousand kilometers per pixel here so if I stick stuff on this now yeah 10 kilometers per pixel I'm going to get a bit of work at the 10 kilometers per pixel level of the pyramid so that I can get you your results fast as i zoom in I'm still working in Maps Mercator because the projection I've got here but now I'm working at 611 neutral or zoom all the way and you know I'm looking at 19 Utrecht is here I can actually say way past the native resolution the data and try to do computation at 1 you get the pixel 1 whatever it is I'm working out we don't care we just do the same work all time ok all right so what I want to do is get you set up to follow along and code editor if you want to set your home computer and you won't be callin 100k you all have our engine accounts then no Jeff will help you get an account and you can go to this link here I can find it push their commercial tender right so if you go to this link we will get you set up with something with scripts specifically for this session only one not definitely time lapse that's okay sign up you can just use GU views on the what 0 1 K is whether those are your image ads are crunching them I've walked up there I think it was like a little over which of them which sugars I care there are you looking to okay so that's good your problem is it ain't right we'll make it work [Music] also Google we can try and resend it [Music] okay sorry see you guys all have a co-editor up with scripts on the left-hand side okay now oh go to back window actually do not okay Kyobo TL / so one kki directing back when you're in jail five zero one kk I'm here okay great now you have just ring us and I shared this you got em she's going next week so you're looking at our code editor the basics are in Foshan left output there on the right to the map reviewing graphical outputs and there's a window for writing code in the non convinced version we actually walk you through building up code from scratch what we're going to hear is I'm actually gonna walk you through existing scripts and we'll play them as we as we go but yeah so one last bit of tour before we jump into some scripts there's a Help button up there under that you'll find a link to our user's guide so when you are having problems or you want to know how stuff works the users guide is a start it's got about 400 pages so there's a lot of content in there when that doesn't work there's a help forum where you can ask questions I answer about 25 to 50 of those questions a day every day but there's a whole bunch of other users that are on that help forum that are also able to answer questions so I get a lot of help in answering this there's a shortcut in the code editor we will cover some of those in the unit there's a feature tour but I'm going to walk you to it directly there's a feedback link so can you send this if you want to send a suggestion or something broke you can tell us and then when you want to suggest a base set when there's some data that you want that you don't have that's how you tell us about data that you want alright so you should see something like this on the left over here is scripts and I'm going to how do you click on script zero and we'll talk about what goes on here so what that does is it loads up out of repository repositories are just get repos if you have built a repository you can click on the little gear icon and see information about that repository including how to actually use it as a git repository if you're the kind of person that uses get you might if you're not then you might want to know about the link that you can share with other people so they can access your repository as well so repos are either read or a writer so you got a team they're working with you can put writer for everybody on the repository and they can you can share scripts this way you can share with your team scripts together because you and your credit is going strong I created this so I have when you're creating your own you'll have your own so quick button script 0 is loaded into the script editor here if we click run it will run but before we do that I want to talk about working here so as I mentioned earlier our code editor runs on JavaScript and we understand that this is a stupid idea the trying to write scientific code in JavaScript is a crazy thing but the ability to do tuning and fast algorithm development we think makes it worthwhile so to be able to use this you need to know a little bit of JavaScript and this is 80% of all the JavaScript Canino there's two more steps that we'll learn about a minute but that's the comment at the top the only thing about javascript is that you have to declare your variables with the bar so this declares that I have a variable named variable and it has a value 1 this is a string variable named string this is a string so you single quotes or double quotes it doesn't matter but you have to use the same one so you can't open with a single and close with a double there are two types of containers within JavaScript there are simple containers and complex containers the simple container is just an array the elements are number to get the zeroth element the first element the second element this array has three elements they don't have to be the same type as you can see I'm one of them could be some other type or something like that it doesn't matter what they are there's just three or four of them in this case now and you get my number the complex container is a dictionary and this is instead of elements being identified by the position they're identified by a name so you give them a name and a value these are called ABCD there's four of them they've got values and one of them happens to be this complex list up here that is built in this thing is the bottom of the function if you used any programming language in the past 30 years you probably use a function the parentheses indicate that it's a function and there's four arguments to this functions those come happen by by position in this case and then there's a caveat about five minutes so if we run this we actually get an error and I did this on purpose because that's what an error looks like in the system you'll hit them I hit them all the time and in this case it says I don't know what number is because my variable period called variable is not called number so I call it number and I click run not evaluating happen what again what has happened is that whatever you've done in this script is packaged up sent off to the server the server evaluates it and the results come back in this case print just says evaluate and give me the results of the string and the string print it into the console this is the console over here so there's our number there's a string there's our list it becomes a button here so we can open it up and access multiple elements so there was a thousand points thousand on the list they wouldn't all just be in one big long chain joint o'clock it would take at one time and then the object is our dictionary and you can inspect down into it as well okay so there you go 80% of all the JavaScript you needed not hard just like every other language and that sort of either you can choose it it's super generic you don't need no funky things it's just a nice ability to use link and the browser runs it so it's a semicolons are optional in JavaScript I tried not to ever do so I'll explain more of that wait a minute the little editor complains that my job is playing semicolon been missing it's informing me that I semicolon Jameson but it's not actually complaining when there's a complaint with an actual syntax error we'll get this kind of thing that says it's not possible for the script to run at all because the Run button is amazing because the script is printable and tries to tell you where your Excuse case I've lost it's uncommon those errors are often weird because it's trying to parse the JavaScript and see what's there and so that I don't go on may be good to close your curly bike-share so that's the error is just expected the curly brace and got something else it doesn't know that there's more but if at least give you the indicator for where your problems okay good thank you right so we're going to go forward as I said normally it is what we would do is we say okay type this stuff and we'll build you up from scratch I prefer the model where we're actually going to look at the script and dissect it okay so we're from top down bottom up so this first script let's just go ahead and run it and look at it and see see what happens so we're looking at San Francisco here this is an RGB composite made from all of the Landsat data from Landsat 8 between 2016 and 2017 so years or the lengths at all composited together being depending on the height account here for four lines of code if i zoom out you'll see that I'm only looking at the parts of Landsat in this location because I have taken my Landsat collection and I have tilted it to a specific location and opening it to only the images this intersect this particular geometry this geometry back hand drawn here see if I comment out that tilt the bounds and I run this again what's going to happen everything will get all the images so the more using that the longer it takes because there's more images involved but we'll let this one percent so there's Hawaii there's Alaska I think it's working left to right here there we go so it's best committed so many motion people working at the relation it's working at the reduced resolution that I'm looking at here and that is 20 kilometers per pixel so that's probably a single tile for everything that image just pulling out you know tributes expecting six pixels actually you can see it's even smaller than that so painting so that's a feel for how this works no actually what's actually going on here so in this case we're working on the Landsat eight collection if you want to find what data we have and what you can do with it you start typing in the search box so in this case I'm going to Landsat 8 it'll take a second to spend and there's all the Landsat 8 we have actually there's more than this even but right now I'm working with the ortho rectify draw scenes if we click on one of these this is so they set description that goes with that data set and it tells you things like band and gain settings and provider link it will take you to more information about this specific data set so that rebuking stuff that kind of stuff so the data subscriptions exists for every data set we have some of them are better than others but yeah it should be the data that it should be the information necessary to use the data so that's an image collection as you already know the collection is just sort of container to contain all the metadata goes with this and we are filtering that by a date range and geometry range now those are pretty self-explanatory v2 diving you though okay land that's one of our bread-and-butter datasets so we actually have quite a number of Landsat specific tools and one of them is simple composite and what simple composite does is it takes the entire code it takes the collection length that you've given this collection that Landsat scenes you've given us computes talk about mysteric reflectance a cloud score for every pixel and then sorts all the images that overlapped by cloud score so this will give you a least cloudy pixel image really easily you can do all that stuff by hand we have those pieces broken out into other pieces but this is a common operation that we made a nice simple function part I want to talk about this here for a second in Python there is a notion of teeny value keyword values see value pairs yes thank you words thank you when you call an argument we call function you can give it a keyword arguments that say my collection is this my option and the motion of the fifth option of this function it has this value again you can scribble items we found that to be super valuable with JavaScript doesn't have such a thing so we've been to wire you can pass as an argument as the first argument to any function a dictionary and just specify the arguments that we care about and not specify 80 hours and we'll figure it out simple composite actually has quite a number in a 5 6 5 argument here and we're only specifying two it just so happened that I wanted to specify the first one in the last one in the process of doing this and I didn't want to fill in values for all the others so using the call by dictionary notation here okay the adds float means give me my numbers as floating point as opposed to scaling left to 0-255 typically the raw Landsat scene Communion and here at 75 and compositing them we want to get me back at 0 25 that's what you wanted but most of the time people want Toa 0 to 1 so that argument just gichan's talk about the strategy of one and then we state that from that image on map and these are visualization parameters to style that English for viewing purposes in this case again this is a dictionary this isn't exactly call by dictionary because it's not the only argument the second argument of add layer expects a dictionary of visualization Kremers there are a number of these but in this case when I specified is a red band of blue band and green band so bands for band 3 and 2 which happened to be the red green bands of 8 and then I specified a scaling factor to stretch my image so the minimum value to the black point of each color band and the maximum value is the white value of the exponent so for red or blue or fill green Toa reflectance looks good at about 0.3 so to get a nice looking image ad stopping noises those were good numbers you can also use well let me discuss with a potential thing on this last perimeter is just a name and that name appears here in the layers panel so when you've got tens of images to 27 inches 20 images stuck in relational it's going to know which ones which otherwise we just called layer 1 they're two and three so that's what the name is for but also here in the layers panel I can toggle this off and on so I can see beneath this graphic image I can make it let more or less opaque with suppressants capacity slider and then I can actually do interactive stretching here in this ephemeral visualization tool so I can actually say go stretch this compute a 1 Sigma stretch that's what those numbers are apply it and then be able to fast visualization so if I didn't know I'm in a max here I can just take whatever the defaults are and then pop into this interactive réaliser to come up with some better numbers this is completely ephemeral I click run again whatever it's done is thrown away unless I import it but you know if I wanted to say oh well it has figured out for me this runs to replace 2.12 then I can just copy this let me back into my script or go through the many ports we're probably more about imports in just a second so can you save these custom Scribner's yeah that's what it for cuts so if i click import they become this thing here and now I can use it so generally you will use Moodle going to UM show let me talk about that per second this section at the top here is called the input section it's just stuff that we got it scripted for you it's just more script but it tends to have some special meaning in this case the geometry is editable so I can grab this and drag it around and it will automatically update you can see if there in the network's things that aren't in the import don't aren't editable that way and I can put data set in here so if I look grab one of these days that I can import it into here and now this is clickable so I can get directly to the basic description so when they just give you some interactivity but they need to be up there imports where you know what they are for them to be interactive we can we have any random thing in strict clickable and then you can pop it open and inspect within them as well so in the case of visualization parameters at the dictionary with these things in case the image collection in a stuffing image collection okay so go to source 6lg um and so what you can click on the point and just start pointless and all those going to the same gia if you haven't done something to cause it to not do that always going to the same geometry so that potentially making points geometry but just back to my sinking script there somewhere and so I didn't actually reach it but I've been using it because the dock is the thing that needs to get think outside optics so there is an enmity map as an object function minute one of the library project to these dereferencing it's and look at the same out of lesson objects that begin with de things in the United States are all things that happen on the server it doesn't begin in the evening space it doesn't happen on the server they're happy going to client so you have stuff for this map here the client this eg aliens a thing to happen block on the server it huge collection as a collection off from server yeah alright so we'll talk a little bit more about some code editor stuff real quick before we move on here in the right-hand panel there's a tab called inspector and what that does is it lets you inspect your map so if you click on the map you saw me doing it a minute ago it tells me my resolution but also tells me all the values of the thing I've clicked on so this case it's a twelve band in eleven band image and there's the value of all eleven bands that I'm displaying even though there's only a red green blue visualization and all it is sort of backing it so I can see all the other things as well okay yeah we'll go on from there any questions yet okay let's go any scripts to we will take our land site collection and we will compute some into the I will go ahead and run it look at it it's an individual image that's a particularly fancy but let's talk about how to get there so as before we've got only exact collection filtered down to the same geometry bounds we are doing a composite and then we have three different ways to compute in DVI so normalized difference vegetation index that is the near infrared band - the red band / The New Yorker red band plus the red band and again three different ways to do it one is to select out the bands that I care about bands formed in five the five is the need to read Boehner today and then do not sort of the hard way I've got two bands you want to go back between bands I have to call add and subtract and divide all these things but but I get my results the second way I can do that is that normalize difference is so common that we have a function specifically to do it and so it will take you numbers of the left side elect part of the equation in the right part of the equation and it'll complete it and then the third way which is more verbose but if you're ever going to do any really complex math is a much better way to do it is with a function called expression an expression actually takes a text description what you want to do extreme and then you fill in the variables down here in dictionary so if you kind of use surface albedo and some creating mathematical expression this would be a pretty horrible way to do it this is not so bad but if you just try to do a real quick one off there's a lot more text students tonight so coming to pick which one we get but all these pretties exactly the same thing okay we didn't sink that one NDVI one of these green DVI layers on the map and it looks like that the questions there this was a single band scaled from zero to one yeah sorry so I think I didn't mention with a simple composite but I didn't mention it but it didn't explicitly point out it simple composite takes the least quality pixel at every point but does it means that any new pixels come to the same time in fact it would be likely to pose ticket for come from same time so with images looking at here we can't say a lot about scientifically we just some NDVI value from somewhere in the year probably not what you want in most cases so what we're going to do next is we compute a maximum in DVI instead of just seeing a DVI thanks so that's screwed three we'll run it maybe I can look the same but let's see how we got there so there's our lint Act collection filter the same way and now what we're going to do is we're going to add an ND di band to every image because ultimately the maximum at every point the way that we do something to every image is through a map and map takes a function so this is the other piece of JavaScript to learn about and how to build a function and they look like this this function is in line let me just just stuck in the middle of another line you could pull this out into a variable like this and then pass that variable into here good sums are equivalent but usually you use this a lot so rather than love X typing in an extra variable you don't need to do that you can stick the function straight into the place where each changed so map is a function that takes a function right this function takes an image and calls normalized difference on that image which we saw a second ago we rename the band because by default normalized difference calls its output and D pronounced difference but I want this will be called leading guy so that's how you reading event and this function returns the original image with our normalized difference added to it at the end so add in senator bands to us okay so this case is that end and in the job convey an image in 11 baby remember once the particular things so at this point we have a collection with every image having NDVI in it and we just run a max over all those so every pixel sort them takes an exit for the dealer otherwise the same at the bottom of this because I have a series that I want that because I have this time series I wanted to show you the charting we have a bunch of broken charts this is NDVI times this is NDVI over time at this point so this is the day of your series it charts as maybe around the x-axis and that's our NDVI over time at that location let's try to interactive you can see click on mouse over it and do stuff if I click on it it pops out and I can do more and can download pieces of it so kind of nice set of charts but we have now mapped a function that makes sense okay that step officially fulfills the EE 101 requirement so in everything now and go to any other session there's more to learn but that's this is what we mean by not being a function and I do this a lot this is a very common operation marriage it's only that when you do the NDVI MX so you take the maximal end or add that actually just took the mechtogan also not yet and then you only then and then I'm specifying one and so I did work I didn't need to do now if the system is usually smart enough to figure out you didn't let alone and so I didn't do them but one thing you can actually do here we're going to be debugging all that here's for con if I create this thing this is an image collection annually and it's got a bunch of it's got a bunch of features in there it's got all those images in it and I can actually stick that whole thing on the map but well what I want to point out of each one of those images has a bunch of an X right I think all the mana I'm paying them but if I stick the whole collection on the mouth instead of printing it I get garbage on the map because I haven't got any visualization parameters but all I really want to do is inspect it and so when I inspect through a whole collection it will give me the value that I clicked on but it will also give me a time series so there's the entire series through all bands so you can see why that might useful if I only want one I can select just the end of the aisle and run it click again and now here at my NDVI time series at that one point so I can pick which band I wanna see in this thing super easy ok this is less impressive now that I've shown you you know this it's the same thing but yeah you can do this for any time I'm debugging anybody script I will print and add layer pretty much everything that God would just inspect few dozen key ok or tear or terrible it didn't work here now how do we figure out what happened between these two steps ok good you play that script you don't understand alright so I assume you can customize the graph I mean yeah there's options not this graph but this one yes you can there's a whole bunch options about what I just said I it's a little tricky to produce a really good publication quality graph here these are more down dirty ladies picked up and report kind of thing but when you want to do really sophisticated you know two backsies two charts for series and this applies not the right this not insult for you but you'd get information like our next battle I mean somehow get the data so you can price bed right you can here just dump it into a chart and then grab it out of the chart as a CSV or something for the chart is just calling other functions that you can call directly so this is a Map Reduce region over the collection at a particular geometry so you could do that yourself and get the same data and just export it I will talk about exports in a minute okay great because more complex you know it's good before this is not all statistics I think people like to do but this sort of is the the kitchen sink script kind of kind of touches every bit of the system so if you understand this well you'll you'll do just fine into the rest of our engine all right so lanes that collection just as before NDVI just as before maximum NDVI just as before but now I'm just selecting the NDVI band which you saw me just do a second ago let's run this so we can actually see what we're seeing and then we'll talk about it so there's three layers here the maximum DVI you've seen open up a big I needed I'm trying to do double statistics then I want to do zonal statistics in two reaches so I just made a regent by buffering my point by 20 Cobbers so there's my region which I've buffered by 20 kilometers and now I need some zones because I'm trying to show though no statistics so I'm going to create some zones out of an elevation map this is the SRTM elevation map I've just pulled it in because I knew it and what I want to do is I want to create a loan for every 200 meters of elevation okay and that's what this looks like here that's what's inside here don't here I was on one don't you something the way I'm going to do that is I start off with a blank image or an image full of DLC and everywhere that my collimation image has a value greater than 200 I'm going to assign a value of 200 in my image of zeros so this is a test and this is a value so I've been ready to test this true accept the value and the value to be another image or it can be a content or as before as I mentioned we do this promotion for you all over the place so this 200 is getting promoted in for an image that is just converted into a multiband image actually it's not Google on benefit but got this promotion chain for you okay so when being there done I have an image of an image that is just zeros 200 400 600 nothing else okay and then let me show you what that looks like and then I am masking off everywhere that the elevation is zero so all the water I'll explain masks in just a second and then I'm clipping that to my region so that my region is clipped so in addition to the circular bit I've got all the water image as well and that's done with masking every band has not just a value every pixel very land has not just a value but also a mask indicating whether it's valid or invalid and actually those numbers in between can be partially dollar or weighted the mask is a 0 to 1 0 being invalid 1 being fully valid and elevation not equal to 0 gives me an image that is zeros everywhere there's water and one everywhere it's not water okay and I'm show you what that looks like by sticking enough to and that's dead right so zeros with elevation 0 1 birthmark people do this with us our team all the time it makes a pretty good krystal krystal mask so that's what I'm asking with and that gives me this result so it's a mask and clicked no water click the circle ok and so those are my domes all right I have a Christian guy i am i interested in coastal zones looking that ocean with a hood and i want to must load means bit of water so you just invert this so instead of not equal to 0 you want to mask where it is 0 ok you're not going to get a lot of NDVI out in the water let's go to work right it's like one two three and four so there's just the water days okay since all right so it's back so click the regions mask with other raster's in both cases you're getting rid of some pixels all right now the actual zonal part that's done here so I take my maximum VDI image and I take my zones and I stick them together so that the NDVI is the first band and the zones are the second band actually zero and one zero band in one bag I'm going to done run a reduced region so I've got a region that might like circle and I'm going to run a mean reducer within that region and I'm going to groove with the results so this is the zonal part for each of my zone each of my elevation sessions give me the mean for just that region I got four zones got four means this is what it looks like when I get it so zone 0 that mean at 2200 400 meters of that one 400 600 years that one above 600 meters about okay so the tricky part here is the group producer I've got a mean reducer and I want to run it in everywhere grouping together the values and so I need to tell it where my groups are so I've got this tube an image then 0 and one group on band 1 define a six-man image of our 10 band Landsat image and I stuck my zones at the bottom and this would be group on band 11 of 10 generally done in by so this is just which one of those bands at the groups and which one do you want to enter meeting on and distraction and if this was all mylanta this would be a this wou Purdue a can band meet lien for ten bands all grouped by this one grouping I'm specifying a scale when you render this region we will try to figure out the scale the projection in order to run at but I'm running on a composite deposits don't have scales or projections then i zoom around you can see that this thing it's run all this stuff it's run again and again and again for the zoom level and the region I'm at that's the all right I said that bomb that's all true but I go with it composites don't have projections so you don't want to rely on the composites projection automatically you want to say run this at 30 meters that's half again years and I just the volcano is always in your cells we just didn't even option it's always movers there are a few places where you can specify a use to something kernels is one of them so if you wanted to build a kernel you can say give me a square kernel in 30 meters or 10 pixels or something like that but those are pretty uncommon and you'll notice that is not scale scale always mean leaders okay then we get printing that result produce region produces a dictionary that's what you got here the one element of dictionary is a group looping and group is a list of other dictionaries each one is a group ending okay alrighty so you want to export that as I excel the next step is export so I actually don't export that export something else but exporting that complex hierarchy in the world trip you know more to flatten it out is something that would fit into an expedite Excel spreadsheet do you like we don't offer erection up you record Geo Gatien and CSV battle isn't either actually it's a new generation but if Amir I would actually flatten this out with some some utility functions to pull the values out of the dictionaries and turn them into lists and kept everything okay export works roughly the same for both images and non images and the non image part is takings so you know an export something that's not image we're going to do it as a table I mean actually the one row table David just mean the one column one row table but it still goes through the table export mechanism so exporting we have our Landsat collection we convert to NDVI with region and export is just a function so what do you want to export so what we can do you want it to be exported in what's the scale and what's the projection if we run this script you'll see that my region is circular that there is no such thing as a circular image so you actually get the bounding box around your region which is the same as ounds we just run down on it so that's what we're really exporting right oops down the toilet I can't put it in the right place I'm just dyslexic today oh yeah so that's what we're actually exporting this bounds the rectangle around our bounds and once you called export they create the tasks over here in our task list you see the task list for doing another task there and lined up so you click run is the last second to change the name or where you wanted to go or anything click on in it común 52 seconds your image will be waiting for you in this case Google Drive you can also export to cloud storage if you have a whole bunch of exports to do like your Drive account is probably on 25 give your cloud account creation Watson Express ESS is excellent back interpret Megan's real so when you create these a like an intermediate or great these new data sets are they yours are they now public are they amazing your accounts eventually in that case I'm exporting to Google Drive that's just Google Drive when you see that achieve unless you share yeah if you export the cloud storage those buckets are by default private until you share them okay so it's a web bucket but it's just a pride of the bucket into your shirt okay what if you get for it for example and here you I mean I export to Google cloud is no we don't care you can but I go for the legal ramifications of getting Khoya data you if you're going to show that data you can share it with whoever it is you want to share it with and they can download that or not okay is it secure yeah does it meet a standard or cloud storage probably meets the whatchamacallit government certification earth engine has not applied so we probably I don't know if we already don't kind of thing called what it was thermos certification all you know you're not exactly the eliminated offenders this is this that's Arnie plated it yeah it's not about no non-federal around right whatever private cloud has gone through most of those processes and I may leave your website ahead ramp dog out you guys plan on going to the rough enough people requested it we mind but also things change pretty fast and so we've got an office effort by having to reapply every three months as we change stuff you know we just added a totally table thing that's a whole new piece of the system that would not be certified by our estimates okay so we're looking at things happening in phases right so that you know the first one is better integration with cloud as old because right now our engine specially developer is quite disjointed from other developer experiences and so having a shared console having a shared developer ID things like that and you know potentially doing billing and other things through cloud that you know then makes it easier for us to do more things like that so that our members certainly not be the first thing we do but you know thing is that the things if you are using other pieces of project structure then you have the ability to use that in a better of capacity with them ready are you yeah of course service for you for service so I learned yesterday that the FIA data has been authorized to be putting to us by staff is something that Sean Healey is our main course contact he told me that they just created pretend to happen okay good BFF a huge thing because I definitely agree to mention such more and I hope valuable them babies these days which to the forces this case treats yeah alright we got one minute and I've got two more examples so they will fast well Justin last night I had negative five minutes and a worth it so I had some time to save half its loss so Italy in it example six some visualizations so this one's pretty simple but it's set up for the next one so this is three layers the red green blue image that we've already seen a false color image which is just changing the visualization parameters and then a thermal image which is a different utilization parameters let's look at what those look like the red green blue is Vance four three two is exactly what we get in there to go the false color is vans five four three New York red red and green so that's a pretty common way to look at education images for vegetation because it makes education stand out you see that I love and then this third one is the thermal band stretched so that the temperatures are some more less than visiting backs of the temperatures which you'll notice if there's a pallet here with four colors in it so previously the near the max were black is the min I like is the max stretch between them and when you do that at the red band as a blue band and green band so I could only got one band and it's a grayscale but that's the default palette black and white the non default palette is whatever you want it to be in this case it's four colors so the minimum blue the maximum is yellow the red and the orange are just linear interpolate it in the middle so you can put plenty colors in here on how to color in here with care or sterility or interpolation through those exam and these images are just stacked on top of each other so you can actually happen between them whatever it is you kind of want to do but often this is a tricky way to look at data and so our last example is sort of a teaser to show you how to build it back in a few lines of code and show you how you might want to visualize data together side by side so we have a set of UI building components this is one of them as you move one of these mounts the other would follow it - those are linked to not component and his window then here is just hard our roof window and by default the room window has a map in it but you can put anything lunch with them so in this case I've cleaned it out and put in three Maps I could clean it out and put in two charts and a bunch of buttons anything and once you're there and the idea is that you don't need me a web programmer El Nino logic JavaScript you already know all the JavaScript you need to know to be able to do this this is a real simple way to build the maps to make it easy for you to do your own work clothes but also to share with your work clothes with other people so I'm going to walk you through this real quick in our negative two minutes so our collection our composite our visualization parameters step together in a dictionary because I want to be able to loop over all of them and do this thing three times I can do I could have done it three times by writing code three times but instead I'd rather just do it by writing a loop and this is everything to build our ass so in ten lines of code we've got our little connected component Carlita macular so what I'm going to do is I'm going to make three maps and collect them into a list for every element in my video relations I make a map turn off all the controls because otherwise this would be a really messing map add a layer to it appropriately visualized stick a label on the top with a name in it and hang on to it stick it into my list I then take my route window get rid of all the widgets and stick in my three maps and then I call this special thing to link up if I don't call this guy they're not linked as I move one the others don't move they'll catch up here in a second actually might even zooms together right so they're completely separate with the linker they it's a special widget that you don't get to see it just clings to get other existing they come in together and stay linked and then I just Center one of the maps and the other would follow so these are tiny bit of code a simple application there's another example in the bottom of this which is a non-trivial application of in creating digital Explorer that got built pretty to you and it lets you explore Sentinel one data and sending them to data I'll show you a cool result out of that just because it's cool so we there but religions this is a bunch of data and what you should be able to see here is Solis changed my settings here so I notice I'm not clicking run here my maps are changing all the widgets the controls over here let me control this map again without ever having to add any more code what you're looking at here is central one radar data through the Straits of Dover for me of our individual ships caught at particular point of time looking through three months of data so there's a maximum of composite waters only bargains to survive these are the shipping channels to the Straits of Dover just popping out its image you want to see how complex you eyes can be built this isn't a particularly well written script but it is you know it'll get you there so that's it we're over time so we can head out but you have any questions I'd be happy to training through anything if you can what I mean it was a press it up in jeans we got em loose session sprinklers the morning I'd have like within India I think is actually I think somebody's actually teaching a tap engine class samizdat 11 items right well there is no agent no data then try to be able to fix 4,000 swing now um if you'll do fight lead it really we don't yet have a way to take knowledge and code and just shout it out okay but everything we do there's nothing special that we do which just we try to make it easier the I'm never the title doctor so we try and piss it here and then II really for people that will is once day but that's done so web apps um when's the tomorrow three o'clock washing you got another web apps as well as some steak is pretty cold steak great Thursday okay so two more times you want me to do some good stuff rejoicing that gives me room now it's very much no no is are plenty of them acquitted yes that's already encoded Google portrayed the way yes there was also finding this horrible okay so I'm not sure what just yeah my dim over sometime are great p.m. is here to give us all the security EVs of poor service Estes probably some of it I have a lot of security to deal with it's not nowhere near as bad I used to work special projects for NASA and whose special projects for Saturday for use on the defense Delta Force intelligence support all those
