Python: Working with Raster Data

good afternoon welcome to our session today we're going to talk about how do you work with raster using Python right okay so I'm normal no additional man I go by no one that's easy my colleague lays here we all work for this special analyst stream at Redlands so before we get started you know just let me see how many of you have actually used special analysts anybody oh why are you here you know everything right how many of you have used arcinfo by any chance quite a few okay how many of you have used ml two hands okay ah how many of you have used grid our confer grid right okay so mostly especially ilex user that's good because we're going to talk about specialist and Python and lustre data okay so um let's get started so the way we're going to talk about it is actually through an example right and part of that example which is a modeling process we will introduce things that we do with roster in Python right so first we'll talk about how you can actually manage roster with tools and you can par for how you can perform analysis using math algebra alright and we will kind of elaborate some of this as we go through the presentations right the first part will talk to you how to access the analytical capability that we have and then how to write from simple to complex map algebraic expression then we'll also talk about how you can actually improve your analytical capability by adding new functionality right and how can you do complex functionality using classes and then from that will go totally on a different tangent a little bit that if you don't have any tool how can you write your own analytical solution actually right and then we will finish off with talking about how you can do roster analysis using multiple core multi-processing right so as I said that we will try to do that through an example so we're going to present somewhat a complex modeling problem that's what you may think but it's really if you break it up it's not that complex right you'll be working with roster features and table and we will be actually writing Python scripts for various reason and we'll go through that now when we go to that you want to learn Python and roster data more than the modeling itself that we are presenting because the objective of this session is not to teach you modeling but actually to teach you the functionality in Python and map algebra you can use to do that kind of complex modeling so kind of you know we'll go easy on the modeling part but we'll focus more on the Python part of it right so why do you want to write a script you're here trying to do Python you have geoprocessing tools you can open and you can do stuff there or you can build a model why do you want to do Python right people do it for many reason some people just do it because they can do it but you know if we actually look back and think about why we want to do it we want to do it because we can reuse it right it can be dynamic because if some of the things are changing you can actually define that as a variable and you can rerun it again and again you don't have to worry about it right you can also extend your capabilities right because as I said we have many many tools but we don't have everything and you want to write something you want to do something and so you can actually build your own - and in many cases it's actually performed very well because it's fully optimized in many cases right so we will demonstrate that through Asheboro model where we will go through different scenario where is actually moving by flight that means is actually flying and going from one place to another or is moving by hitchhiking through roads campsite mills population and other things and finally there is another step where is randomly moving so we will try to simulate all this different scenario in Python so that you can see how a complex model can be broken down into pieces and then solve through series of scripts or vital settlements all right so now I'm kind of going back to the beginning again to do raster analysis why do you do raster analysis and what are the things you do when you do raster analysis the very first thing of course you want to prepare and manage your raster data right so you want to display you want to add something copy delete all these things you can move like you can clip now to do all this thing you will use something called raster object and we'll talk a lot more about that sometimes you'll use numpy you also you have apply environment that you can use to do it right to perform the analysis itself you will rely on special analyst extension to do it but the way you will do it is not by opening the tool but you will write map algebra so it's algebraic expression that you're going to write and you will do analysis in that manner so what is map algebra now pretty much everybody understand algebraic expressions people explain it in a different way but that in simple words is a very powerful algebraic expression through which you can actually execute special and it's tools right it has operator setup functions and your performing geographic analysis so you're writing an expression Python expression which is doing geographic analysis right the utilizing the tool in specialist the strength in creating complex expressions you can actually create a very powerful expression in a very short form right it is available through special analyst module and it's integrated in Python that's why we are talking in the context of Python right okay now how do you actually access this map algebra right there are many entry points in our system right the first entry point is raster calculator even though is a tool is a dialogue but you can write map as a way algebraic expression there right so it's a specialized tool it has easy to use calculator you know interface if somebody used special and it's extension from 800 use rostra calculator is same as this one right it's a geoprocessing tool and it can use it standalone open up the tool and use it or you can use it in water builder right the objective is to use it either in this two place not to really call this tool in Python because in Python you will be writing expression by itself you don't need roster calculator to write that expression you can also do this expression in Python window you bring a Python window and you can do it and this will show some of these things to you you can also write your standard script and do fancy programming there right in order to do all this thing the very first thing is you have to import a module in Python you have lots of module you know packages so in our art PI package you have we have a module that is our special analyst model right like every module it has to be imported now we don't want you to import in anyway you like Judy you can do that but there is the advantage of doing the way we are saying where you import art by and from art by import environment and then from art PI dot sa import star now if you're a hardcore Python program are you going to like you will cringe to it I said why am i importing everything just bear with me you will see that by importing in this manner you can write much simpler expression you don't have to qualify every function you use and that way that your expression become much more simpler that's why we recommended this recommend this way when you are doing that of the algebraic expression now what is the general syntax for this right the general syntax for writing is is very simple right your output goes in the left-hand side and your tool parameter and operator all goes in the right-hand side simple left-hand side is output right-hand side of business if you are familiar with doing geoprocessing tool in Python that's not the style they use that that those tools core tools eyes actually have result object in the left hand side and everything else including the output on the right hand side so take this take note of that and you'll see that in the example it just in the in the following example we are saying from our PI dot si province from our PI dot s a input star and then out raster that's my output equal to slope and my input so you see output on the left hand side input on the right hand side right and this expression comprised of input data operators tools and parameters and finally you have output right what are the common inputs for analysis rosters features number text we also have few more things objects constant variables now if you look at this expression at the bottom it has your raster it has string it has number okay what are those operators we all know operators or plus minus and all the mathematical operators or operators right now the same operators exist in Python and in map algebra or in special and is right what is the difference right let's see out roster one equals two in roster one plus in Rusted - what does it say anybody is adding to roster right but if you are doing Python without roster is adding two numbers or two variable or string is something else so depending on the context is doing something different in map algebra when you are doing specialist and you are using the same operator plus it's actually adding to roster together right now sometimes it doesn't know that whether you are actually referring to a roster or you are referring to a string because you want to specify the name so to avoid that confusion not only confusion it actually won't work so what you want to do you want to actually cast your roster as raster and you have seen that in other programming right if you have an integer you do int you have flawed float you do float so it's cast you cast that thing as a raster and then you do your operation so at the bottom you can see out raster one equals two raster then rust and m1 if it is the name then plus roster name - and you are adding them together there are other expression to show you that you can actually add a roster with a number right the most important example is at the bottom you see it is using the output of the second expression and then adding something else to that this is where the roster object becomes much more powerful and we'll talk about that a little bit more now when you are adding something to a raster object it is already raster therefore you don't have to cast it as the roster anymore so you see the out wrasse - we are not casting it as a raster because it's already a master but the raster name - where testing it as a raster because it's just a string what are the tools here we have in map algebra tool what are the map algebra tools so all the specialist tools you see in the toolbox are available as your map algebra tools in Python in the model right you can actually use all of them right you can also use any other Jew processing tool right but remember the to sort get sensitive right now if you look at it how does it look right in your tool dialog you have input output another parameter and when you actually define that in map algebra is pretty much how the tool is executed right it has unique set of parameter some are required some are optional and some numbers strings objects and when you actually write the expression you see your output and then slope is the name of the tool your input and then the other parameters going there so pretty much same you just have to remember to move your output on the left hand side alright what are the output the output is the raster object which has methods and properties and the most important thing is the output is temporary in nature okay as temporary in nature if you don't save it it will go away right with that I'll hand over to Liz so that she can show you some of this things in action safety no man as Naaman mentioned my demos today are going to have to do with a beetle called the emerald ash borer and this beetle is extremely invasive it attacks ash trees it just kills them and my study area is in Vermont so that's the red box and the data that you can see displayed in there is from land use data not in this demo but in my next three demos I'm going to use some campsite data from lumber mills an urban area and some major roads in this demo I'm just going to briefly show you the Python window in the raster calculator that no one was speaking about so I also have some elevation data the first thing is this is the Python window down here at the bottom of my map I was able to enable it by clicking on this Python button it's just a normal IDE that it executes the Python one thing you should know is that the import statement import arc pie already occurs when you open up the app but I just want to remind you that all of this is available because we've already imported our PI as nomen mentioned it's really important from our map algebra that we overload overload those operators in order to do that we have to say from arc PI sa import everything and that will help us overload our operators so we can do math algebra now that I've done that I'm going to go ahead and create a raster object so I'm just going to say a live equals raster and I'm going to select one of my layers I'm going to select this Vermont elevations layer now I have a raster object called alev that was create on the left hand side now this output is temporary by Nature and it does a lot of really neat things one of the neat things is that off of the raster object you can get all kinds of properties I don't know if any of you have ever used the described method in Python or the get raster properties to your processing tool you don't need that anymore if you have a raster object you can get all of that information right here which is super awesome I think it's way easier and there's just a huge list of them there you can briefly take a look at them but go back and reflect on them later I just want you to know they our cell size yes right there you know just really handy and easy to have and very accessible the other thing that we do when we're doing raster analysis in particular is set some environments upfront so to set the environments in Python Arkady and V dot and I'll set the extent here to my study extent nomen mentioned that the operators are over overloaded which allows us to combine operators and values along with our raster so we'll go ahead and see an example of that now so my elevation data set right now is in C and I want to convert it to meters which is super easy to do because I already have my raster object and now I can just multiply that by a conversion factor not that conversion factor though there we go so now what I've done is I've created a new raster object and which points through well a temporary raster and it's now has the unit's metres nomen also mentioned that we can access all the spatial analysts your processing tools via math algebra such as slow so we want to see what that looks like we can say slope o equals and you can see I got some helpful intelligence there that helps me fill in some things which is a really nice aspect of the Python window as well as all the help down below what I'm typing which is nice so I know that my first input is a raster but in this case I'm going to actually use the raster object I just created so that's my first input and that's the only one actually me the others are optional and right now I'm running the slope to generate a slope of my study area like Naaman said we don't have to do it in a Python window when we're in the application there's also a raster calculator tool and it lives inside the spatial analyst tool box inside math algebra inside raster calculator is the tool and this is where you can enter python syntax so i've quickly done what i just did in the python window and just this tiny little model here so i have my input elevate Dadda then I used this raster calculator and applied the conversion factor and then I just ran the slope tool just to give you an idea that's how it's accessible in the app so we talked about you know how to do our import statements we talked about the raster objects we talked about some properties we talked about some environments as well as using an operator and a function and I'll go ahead and hand it back over to nomen now Thank You Liz oh she covered a lot of things very fast right but if you actually look at them carefully these are pretty simple things to do right one of the most important thing she did is she used raster calculator in model builder in a mortal now if you step back for a second I said over a second you have all this plus - all these tools there right why can't I just use one after another and then do it right of course you can you can bring times and do the multiplication you can do plus but think about it if we're doing a times B plus C times D or something something something then suddenly how many tools you have to put together it becomes really cumbersome but you can still do it but if you actually use rostra calculator your expression become much more compact readable you can actually look at it immediately see what's happening but most importantly it's actually optimized I'll come to that in a minute okay so moving on let's see complex expression so she did simple expression now we'll kind of do a little bit more complicated thing where you can actually combine multiple operator or tools together to create your expression and still very readable right you can it can braid it now you can have an output from that complex expression and then pass it down to another one now you don't have to write a super long complex expression but you can actually do a sizable var and take the output and pass it to the next one you can logically take it down your workflow and do it this way it's actually good practice to do do so right because then you can debug it easily and you can actually think about it more logically right if something goes wrong you can execute one statement at a time and see where things went wrong believe me things goes wrong all right okay and I'll more on Ruster object Ruster object is very critical for map algebra and processing with raster right it's a variable pointing to a raster right output from a map algebra expression that could be a raster object which of course created raster or you can actually point to an existing data set the way Liz did first she created analysis for object pointing to the DM she wants to work with and then multiply that with some numbers and stuff like that now the data set that is associated with this austere object or the raster object are pointing to is actually temporary in nature when it's actually the output of your expression if it is the data that already exists there of course it's permanent it's not temporary but the one that's getting created new one is temporary you have to call save to save it now you can argue and you can say why do you have to do it why is disappearing well think about it if you are actually doing a complex model and you are creating a bunch of intermediate one either you delete it clean up and hang on to your final result or we can do the job for you so if we are doing complex modeling I think we thought that if we do that job for you is actually better it's not going away immediately it's going to go away when you are done with your session or you're killing that object or something so if you are adding the output to display you can still see it it's not that it's somewhere magically disappearing right it's disappearing when you are done with it right and series of properties are there please already demonstrated that for you right and that's actually a good way because on many occasion you will need to know the number of roles you may need to know the extent you may not to know the mean or max value because you have to make the decision and those are actually dynamic right so if you actually can create and use it in a program if the whole thing becomes much more dynamic right I mentioned about optimization right before we talk about optimization let's talk about local tools like class if you are doing class it's happening what SL is added to another cell is local operation right some of the operation that if you are doing a plus B times C something like that think about it you don't really have to do it one by one what we can do we can do all this thing in one shot right so that's the optimization we do and because of that you are not running one thing at a time you are running everything together in that batch algebra expression things become much more faster for you right does the optimization when entered into a single expression all local tools and operator processes together on a Parcells basis which means we are optimizing the heck out of it right okay now we're going to go back to our complex model that we started to introduce and we will actually show you Liz will show you how to similar the movement of this ash borer through hitchhiking okay okay so before I move on to that no man mentioned that the raster objects on the left hand side is temporary so you can see I queried it here and it's true it's temporary I just want to talk really fast about how to save that result so there's a save method off the raster object and if I just execute it like this it'll execute and it will persist on disk it will it will save it'll make it a permanent raster to my trash workspace for those of you who you should look at the environments but scratch workspace if I give it a name it'll go to my current workspace it'll write it to my current workspace or you have the option of giving it a full path and I'll write this with the name you provide at that point so that's how you use the save method to persist your final result as nomen said I'm going to talk now about how the ash borer moves around and I don't know if any of you guys can but sometimes at campgrounds you see signs that say please do not bring in firewood do not take the firewood here to another camp and the reason is sometimes that firewood is full of this invasive species and it's one of the main reasons of how it moved from place to place throughout an area is either on lumber leaving lumber mills or it's in firewood the campers and advertently unfortunately move and continue the infestation of this beetle so let's go ahead and look at this hitchhiking model so it hitchhikes when it's on wood and it's being transported not by flying itself there right so I don't know if any of you have done any type of suitability modeling but the first step in solving this problem is kind of figuring out where can this beetle get from if it's in a campground it's going to it's more likely to be closer to campgrounds closer to roads closer to urban areas closer to lumber mills and so we're going to kind of create a surface to help us solve this problem about how likely is it to get to one of those locations so what's important for something like the campgrounds is the distance to the campground how close the closer you are to a campground the more likely you are to find the fetal because that's how it travels so what I did was I know that it's not a linear decay function it's not as it moves away it becomes on equal scale less is a nonlinear decay function so the first thing I did is I took the campgrounds I calculated a Euclidean distance and then I found the largest distance within my study area just simply so that I can normalize it notice here I used get raster properties because I can't really access the raster directly to get the properties off it and then in this raster calculator expression is where I normalized the distance so I divided it by the mass to get it from 0 to 1 and then I applied a nonlinear decay function as it moves away from the campground how likely are you to find it really likely next to the campground much less likely the further you move away from it and I did this for each of my different inputs to my kind of suitability modeling here and then I combined all those in a raster calculator which is another complex expression which is what we're trying to talk about here so I summed up all these rosters and multiplied each of them by a weight and then divided by the sum I'm going to show you all this in a script as well but I sometimes a model is easier for some of you and the scripts easier for the rest of you so just to kind of get a picture of what I'm doing I thought I'd use this model first so after I have a raster that represents how likely it is for a bug to hitchhike there now we need to see how likely is it that it can infest that area right is there actually a tree is there actually an ash tree at that location otherwise it's just going to die it doesn't have food so that we need to consider that next so I took that likely how likely is it that the that the beetle ends up there and I put and I said okay if it's greater than 0.5 then it's going to be likely that's what I'm considering likely remember my scale is 0 to 1 so if it's a higher value it's likely to end up in that location so I use Kon to do that and then I just did some intermediate steps here of converting it to a polygon and dissolving the polygon and then I created some random points in the locations I identified as likely and then I check to see if there's ash under those points if there's ash I'm going to say the bug is now living there it's infested that location then I'm going to loop through and run my model or my script again to see where those bugs then hitchhike to and that's how we can simulate the hitchhiking movement of the emerald for beetle so here it is in a script the first thing I do is my import statements as you recall seeing in the Python window then I set some environment settings extremely crucial for all raster analysis then I set my parameters and then I actually get into my loop of actual crunching data and doing some analysis the first thing I do down here is you can see that I run my Euclidean distance tool with the with my first input and now look I can get the max data just off of the raster object directly so after I have that maximum distance then I'll be able to normalize my data to keep it all in the same scale because like you can't combine apples and oranges you know so we need everything on a scale of 0 to 1 so that we can sum them up in that next raster calculator statement so down here you see that expression I had inside a raster calculator now it's just directly in Python right that's my nonlinear decay function and I'm multiplying it by a weight maybe their location the campground is more significant than the location of the urban area for example when I combine them I get to weight them in there in terms of their important nasir you can see my con expression where I said if it's greater than if it's if the likelihood is more than 0.5 then consider it as a location to be infested highly likely to be infested and then down here I took the raster that represents where the actual trees are that could be infested the susceptibility and then I combined it to say okay well if if the beetle ended up there and there's a lot of ash there then create a newly infested area and then I loop through this a few times to have that location grow out so if I come back here and I turn on so you can see there's a few spots there already but my script is running right now and if i refresh this this may take too long if i refresh it you'll see in a second more pop more locations where the bug has hitched egg to will draw on my map because I'm adding to the existing raster there they are and it'll do that for each iteration because I'm growing that out so in that example I hope that you take away some of these complex expressions where I put multiple functions together I used functions as well as operators to do some of this analysis like this and it's really great because it doesn't write intermediate output so like there's no there's no output created by this division and after that's there then it doesn't create an output to when it runs the power function and then create an additional output when you apply the way it's kind of like a deferred evaluation where it executes set all at once so there's weight fewer read and writes when you're using these operators in math algebra and that's why it's really great to be able to combine them all together and a complex expression like this or one of the reasons why and I'll hand it back over to Nolan Thank You Liz a little bit of complex expression but it's all good because you will have actually this sample and you can try yourself that how we're actually putting this thing together doing the looping doing the complex expression now operators are simple some tools are very simple samples are not like you know for example neighborhood right if you are doing some kind of neighborhood operation you may have to specify neighborhood right and anybody has used focal statistics by any chance anybody a few people here so you know you do that in a tool dialogue use that's why something but question is how do you do that when you are actually in Python to make your life easier we have created something we call classes it sounds difficult but it's believe me it's not really right so basically these are kind of a parameter that we are going to use in your Python expression aha it's almost like a small tool by itself you can think of it and it has a bunch of parameters you just have to specify a bunch of parameters and depending on the classes the number of parameters varies right and also depending on the type the parameter could vary right so you'll see that example of that what is the benefit the benefit is very flexible because you can easily specify the parameter that part is flexible the most important so X ability comes from the fact that you can query it because if you run it GP to Lear on you go away and suddenly did I use five or seven right Oh what happens if I use five can I try with seven then you have to open the tool do it again but if you're writing script you can do that very easily you say certify five and say what did I do or five let me add two more 7.now let me add two more nine now so you can do a lot of what-if scenario and modeling using these classes very easily right you can create it like that it's a neighborhood class neighborhood circle and of course it requires your radius and the units you give that and you can actually now request it back you're querying it and then you are changing the argument and actually increasing the radius by two simple right another thing about complex expression that when you are actually doing roster analysis sometimes you also have to incorporate feature in your operation you may want to create a surface first and do roster operation so you can actually combine all those tool together here's an example where you can actually use IDW to bring in freaking data and then take the output and actually do roster operation you can bring in any geoprocessing tool you want in your expression right so we're going to go back and show some of this complex nature through another example is we'll show you the movement by flight where it she will use classes and show you some other example okay so we talked about how the emerald ash beetles can move on logs and firewood and stuff like that but they fly too they have wings right so that's not their only source of movement now there's some things we need to consider I know for example outside of my study area to the south all those ash trees are gone so I know that these beetles are only moving north so I need to be able to control the direction that they're moving not only that I know that they they don't move at all in the winter you know you never see beetles flying around in Vermont and the smell right so like how far do they move in the spring how far do they move in the summer and how far do they move in the fall I want to be able to change those and model those things so this is the idea again I'll show you in the model first I have no one locations of ash borers are those yellow dots and then I use the focus statistics tool to control not only the direction but the distance of where they could possibly move from there know one location and what I did is I specified what's called a wedge neighborhood and in this case I sort of set it up to look more like a semicircle than a wedge because I said for 180 to zero is kind of the direction they can move which of course would be north because it's everything in between and then I said the radius is cell size of 20 they can that's how far they can go the way I have the tool setup here and then after I see where they can go then what I do is I see are there any ash trees there and if there are as trees and infest those ash trees and then I take that answer and I concatenate or add it back on to the rest of my infested areas and rerun that model and I can loop through and I can see how how it moves through flight and where the infestation goes in a season or in a year so let's go ahead and look at the script here so again I start with my import statements my environments in my variables here you can see that I set three different distances I said I have a five to represent five cell size for the spring in the summer they go much further I say they can go a distance of 20 cells and in the fall I say they can go 10 cells I should just I'm not a biologist this is made up to teach some math algebra so if somebody knows more about beetles I hand it over and direct all those questions towards you but that's how far I'm saying that they can move in each season and then I go ahead and I start my geoprocessing loop kind of down here where the real work gets done and you can see here this neighborhood wedge that's the class maybe I'm going to back up one second this may help if I come down here to the Python window and I say a equals focal loops focal statistics so this is the tool right that I'm running it's just implies on the first input is a raster I'm just going to kind of cursor through that and now you can see the next one is the neighborhood and if I go down and select weight it's a class that has inputs and the input to this class are the radius the start angle in the end angle and because we've made some classes and not just complex Springs they're strings they're extremely easy to iterate through and set different numbers into each loops that I loop through them and that's what I'm trying to demonstrate in this script so here I've created this neighborhood wedge and the distance I'm getting each time I go through my loop I'm setting it as a different value either a 5 a 20 or 10 to see how far it goes each loop and again this is how I'm creating the movement only to the north and then I run the focal stats tool after that I use a complex expression where I take that result and the susceptibility raster which is the location of the actual trees and I figure out whether or not there you can inhabit that location so when I run this script this is what it looks like so not a photographer either so this is where it starts there pixels not points so they're hard to see but that's where they got to in a spring so relatively short distance but then in the summer that could go much further and you can see there's a contrast between this one in this one and I'm guessing there's a lot more well I know there's a lot more ash in this area where there's no ash here so they didn't sustain themselves there and then in the fall they were able to go a bit further but not as far as they could travel in the summer so that's one example of how I used a class to iterate through my loop and set different values very easily and excessively in math algebra Thank You Noah Thank You Liz see the the beauty of this demo was that she used classes to simulate a very complex scenario different season distance and all this thing and I prayed to it with few lines of code right ok now we're going to switch gear and totally go into a different kind of in a different side of the fightin any of you have heard of something called numpy numpy whatever okay good okay with the most common thing between a numpty array and raster anybody anybody yeah good right okay thank you so you know he kind of explained you know what if the luster an ampere is the most common thing that occurs to me is they're all a bunch of cells with numbers right so if you look at it last year with all this color you change that to number that becomes added as an alum pra all right basically and if you can transform that array you are actually you can work on with them right so to make that work easier for you what we have done we have done some work so we have first incorporated numpy with python so you install you can get numpy you can use this numpy to create custom functions right and custom tools there's something that we don't do ourselves we have not shipped to you but you want to simulate something you want to do the different calculation now it comes at lot of additional functionality clustering filtering linear algebra optimization Fourier transformation morphology you know you name it now how does it work really in our domain basically we give you very simple two functions right one is to create a numpy array from raster it's called raster to numpy a simple that means I'm taking a raster turning into an empiric the moment is still numpy array there you go you can have access to those numbers and you play with it you do whatever you want you don't like the slope we provide you complete your own slope all right that's the idea now once you're done manipulating this number creating your own fancy slope or in a very complex operation you have this number stored in ampere you want to bring it back to your a step so that you can do other operation to it so how do you do that you call numpy array trusty now in the first one you see the input is raster in raster and all the other things are optional when you see this all discarded that means parameters are optional basically if your roster becomes too big you cannot put it in one array you can just specify your starting point and you can just drop it off right that's why those options are there but you don't have to use it if your roster is manageable and you can load into the memory is fine on the other side when you go from numpy are a trusted your input is of course an array and then again all the other things are optional right so once you do that you can do your operation and you can do some fancy stuff and Liz will show you how to do a fancy stuff here okay obviously don't have a geoprocessing model of this because numpy array it's hard to see in that format so I'm just going to go ahead and show you the script here you can see my import statement and this is going to be super neat I import another module this time one called random and I'm going to use that because what I want to do I guess I should start with that is calculate some random movement for this Beatle so either not hitchhiking and not flying but sometimes the models and algorithms I have developed my other two models you know the biologists are happy with them but they say well sometimes they end up other places through other ways well and you know we want to account for some of that random or how does that random affect the outcome of your model to kind of test its limits so sometimes you need to introduce something like random into your into your modeling and how do you do that well we don't have a tool for it in the geoprocessing tool is a spatial analyst to do that but if there's something you want to do and you in it exists in Python you can turn it into an array and then access that module so what I did was I imported the normal stuff as well as random then I set my environment settings as I always do and I define my variables as I always do and then down here is where it gets really fun this is what nomen was just talking about I use the geoprocessing function the raster to numpy array and as input I put my existing locations of infestation as well as my susceptibility raster meaning are there trees there if the bug lands there will it become infested so very similar my other models it's just a different technique to get some randomness into it the next after creating those arrays how do I use those arrays well I created a function here where I found a subset of myarray and then I wanted to generate a random movement distance and that's how I use the random function down here you can see off the random module I use these exponential variant so the point is I'm using something outside of ArcGIS outside of Ezreal sided Pro outside of everything we've done I'm grabbing a Python module and integrating it into my analysis because someone's already written it someone's already come up with it and it's something I need to do that doesn't exist and what we provide to you and one way to access that is through the numpy array and then I can access this random module and apply this to it and it will return like up to a max distance a random location of where the emerald ash borer could end up and then I find out of that new location is susceptible to infestation if it is I add it to the current infestation and I can loop through adding these you know random locations so if I do it once it's there and then another time just a random point within a certain random distance shows up and then likewise the third loop through more points show up so the real point is you run raster to numpy to get your numpy arrays and then when you're done doing your analysis you just turn that back into a raster by using the numpy array to raster function to export that array back into a raster that you can continue on to use in your geoprocessing modeling and project Thank You Liz did you write of maybe not but you know you get the point that you don't have to be limited by the tool we provide to a complex modelling you can start with very simple and you can actually go all the way to very complex the good part about Python is not only numpy you can actually import many packages how many packages are out there I cannot even tell you know and you can do a lot of fancy advanced analysis with it so our framework actually let you do that and you can actually turn into a geoprocessing tool and they all work too I test the power now I'm switching gear and actually want to talk about something different I will just introduce the subject here and at 4 o clock and musket B there is a full session going on they will talk more in detail how to do multiprocessor use multiple multi-core and do multi processing using Python for vector and raster so if you are interested you can go there but I'm gonna introduce some of the core concept that you need to understand if you are trying to do this with rosters because we are talking rosters here now step back and think about some of the function that we have run we talked about plus or multiply those kind of function those are local function operating on a cell right then we have talked about focal statistic there is a focal function that for to operate on one cell I have to look at my neighborhood right that's called focal function there are other functions like zonal function anybody has used the neural statistics okay so that's the zonal function that means your operating on your entire zone right and then global function is something that in order to compute something we have to look at the entire roster those are global function like watershed distance those are global function now multi processing is actually easy very very easy when you are doing local processing then focal go to zonal is doable global very difficult right but the fact that a lot of operation you do are in nature of local and focal sometimes zonal those are actually very easy to do right here is an example he says pleasingly parallel problems sometimes we say embarrassingly parallel problem right that we can do but we don't do and we should be doing actually in our GS now lot of tools use multiple cross processor right zonal stat does we are we're going through it and we are doing lot of them you have heard about raster analysis RJ's image server running on cloud paralyzed right we are doing distributed cluster so this is the area we're going why because if you look at your data set it's not small anymore right those days of where you know 90 meter data or even you know 500 those days are gone now we're talking about submitter level right so the resolution of data is really high even though if you're not processing big area you are actually processing significant amount of cells right so here is an example if you want to process that entire roster in one shot you can do that it will jump through it use one processor but if you chop it up into four pieces you're doing times our exponent or something a square root in this example you can chop it up into four pieces and send all four pieces to four processor and then bring it back done right so these are very easy to do if it is a local function right here is an example what happens to your time so look at this first one it says one there and on the y-axis is the time so if you are using your single processor the time is depending on the which function you're running is like 400 + or 450 seconds the moment you divide into two pieces it comes down to 250 seconds boom almost half you send it to 3 processor suddenly is below 200 you send it to 4 processor is 150 you see how drastically the time is actually changing but then is flattened out what's really happening so the reason you know somebody might say why don't I get more performance out of it because there is a cost here the cost is chopping it up killing it sending it to your processor working bring it read back so if you look at your workflow you have to do certain things here right and when you have to chop it up more more more you are doing some kind of pre-processing those are overhead putting them back over it so at one point the benefit you get out of execution is diminished by the overhead that you are doing so that's why is going down and then leveling off for is a good number depending on the type of operation you are doing so we did this a simple example for a local function and a focal function it shows that for is probably a cutoff line beyond which you are not going to get many Minutemen you any benefit out of it this is a good example to look at here is an example code example here that somewhere in the middle you can see it's a pen depend depend and then somewhere it's a spool and then is sending it and then bring it back only two statements does all the magic right so if you want to actually see how to create a simple execution task and hand it off to pool to do multiple processor you can go to actually leave that session at four o'clock musket beep and you will learn a lot about it believe me it could really change how you actually operate and how your work is done how fast it is done especially if you are familiar with moves like think about it if you have a large roster that you want to process chop-chop-chop and there are tools in and they will demonstrate that you can send and say ok process this extends this system this extent and create all this bunch of output and then you don't even have to put them back together physically you can use mosaic which will do that dynamically for you so the workflow become much more simpler and efficient and you can get a lot out of it ok with that I'm going to summarize what we discussed here when your problem becomes simple or complex especially more complex you need the capability provided by map algebra your life becomes much easier right map algebra is powerful flexible easy to use and integrated into Python right this is for the people who did map algebra outside of Python in there some other life right XS through rostra calculator Python window model builder right and scripting we have talked about Russ object your best friend and process right cluster object if you work with raster raster object is your best friend you can actually create models that can capture interactions of phenomena right have a complex interaction right you can also use multi processing to improve your performance where applicable where applicable is important because you don't want to try to do global function using multiples are not going to work focal you have to be careful because of the overlapping area I did not mention that they will mention a little bit but you know that focal has when you are actually doing the edges there is some overlap needs to happen on how to manage that so there is a little bit trick to it but still very very beneficial and with that all this demos that everything will be available over there you will get that slide so you can try and have fun with that thank you very much we'll take some questions any question so the question want me to take it okay the question is I'll actually paraphrase in this example all the rosters are on local disc the gentleman he is asking that can you run operation where your roster is on disk but your raster is as a service in particular in our cases image service right that has pixel map Celine's doesn't have pixel right image service the answer to that is yes all spatial analyst tools and in my pal jablow Python it takes raster in any form or shape whether is the data set on disk or as a surface so in just a second I think there's another gentleman had another question right so if I understand the question the question is what is the best way to some long list of roster so you have multiple rosters then you want to sum them up right did you try a self statistics there is a tool that called self statistics you don't have to do a plus B plus C plus D plus E Plus F like that you can actually bring up the self statistics tool and send everything into it and say give me the sum done well if you have one thousand and for some reason cells does statistics does not take 1000 I don't know that it owned it probably will work very fine if it doesn't then what you do you do 500 500 500 then add them up again you're doing in this sum right so you can do first 500 some second 500 some so self statistics four times then one times you're done right so you just can nest it a little bit and then it will work right if not come by oh I'll show you if not the beginning of my script just workout me learn in my excellent question and comment right so the most of the image service that you access are probably huge in extent and it has different resolution and whatnot so the trick is you specify your extent correctly you specify the surf size we want to work on right and that way you can actually control it's not about that way that tool will fail there is other limit to it because the server on the other side has a limit how much pixel is willing to give out right so you cannot really get all the pixels you want it has the limit so if you try to access that server will refuse to give so you're running operation it will not get any data right so because of all this limitation it is important to set the extent and cell size so the number of cells number of pixel you are actually pulling through server is actually allowed by your server and then you can chop through it [Music] yes yep you can do that you can set it to a raster you can use the numbers that represent the X min Y min or xmax ymax just okay so so the it will figure so the questions that you are actually making some really good point can I set when you're doing your Australis can I set extend from our features the answer is you can set extend from any geo data set right you know where the feature roster doesn't matter it can be of any projection we will resolve it for you we'll figure it out right we'll do the on-the-fly projection and all this thing but let's be our of a couple of things right when you are trying to mix things up cell size projection you are actually in evidently triggering this sampling so when you are changing cell size you are triggering the sampling when you are actually doing an extent which is not lining up with yourself is moving this way that way a little bit you are triggering resampling so what I recommend is actually strongly we all recommend set your extent set your cell size set your best friend snap roster that's your best friend snap roster how many times people have said I have done this operation result doesn't look right what happened what happened is it happened resampling on the fly you did not notice so your value is now falling on the other value that you did not intend now you're getting wrong result right yes sure you know okay so just to demystify first snap roster has nothing to do with the cell size itself this snap roster you can think of it an infinitely large mesh rate and you are processing a roster only thing we're trying to will match at that point is your left and bottom boundary and line up with that infinitely large mesh your snap roster is think of it as the infinitely large mesh and the area you're processing will move it slightly so that it's kind of aligned with your left and the bottom once that one is fixed then you can have the same cell size different cell size does not matter we'll chop that up top and then we'll do the work right of course of course and you can do it for your snap roster you really the way I do it wherever you are doing a project you have a roster that's your primary roster that you are going off off right and it could be elevation land-use something that is a primary duster everything else is revolving around that I usually use that as my self size and my snap roster and then everything else builds on top of that right okay good questions comments thank you anything else okay thank you for coming up for lunch it was great thank you [Applause]

Channel: Esri Events

Views: 9,042

Rating: 5 out of 5

Keywords: Esri, ArcGIS, GIS, Esri Events, Python, map algebra, geoprocessing, Spatial Analyst, ArcPy, object, classes, raster

Id: aQfci8AvUqY

Length: 63min 2sec (3782 seconds)

Published: Wed Mar 29 2017