ArcGIS modelbuilder tutorial - netCDF bonus 01 | burdGIS

[Music] I'm at fans and today we are looking at more model builder things so off the back of the model builder series that we've been running recently Derrick got in touch he's working in Ghana you can see garner in the background here just the outline of the country and he's working with something called netcdf files which are multi-dimensional files I'll talk a little bit more about that in a second but let's have a look at this model he wanted to iterate over YZ netcdf files that he's got think Derek's working with about 10,000 of them so this model could take forever to run but it is a perfect implementation of why you would use model builder rather than trying to process 10,000 files yourself because you would probably die before you got to the end of it so that models run we can see what's happened we've got a couple of layers here I'm not sure if they're the right ones let's just have a look in our gdb down yeah beautiful all right so this model looks a little bit complicated but don't worry we're going to take it through step by step and I'll show you some tips and tricks as to how to get the model working exactly as you would like alright so without further ado let's talk about netcdf file over at the ESRI site you can have a look at this section of the docs so we're in the netcdf different data type and the fundamentals of netcdf data storage so they've got some nice images here you can see how netcdf is used to store multi-dimensional data so we might have for example temperature over different times we can see that that's three dimensional so it'll be stored as an array also you could have a fourth dimension so it might be temperature over time at various different altitudes and they'll be stored as a kind of array of arrays we don't need to worry too much about the fundamentals netcdf for this demonstration but if you do need to iterate over a lot of netcdf this will certainly get you started so let's have a little look at these netcdf files that Derrick is sent through and it's a good idea always to check out your data before you get started with anything so I'm going to go into the multi-dimensional tools sounds exciting doesn't it and have a look at maintenance EDF raster layer so if we open this to look it's going to ask me for an input netcdf file once it direct sent through are actually in here the files and they've got this file extension of dot NC so it looks at the top here first up that we have something like a date so 1985 first of January and midnight by the looks of it not sure what the IDI is afterwards I don't know what the data source is so I'm just kind of working backwards here so let's bring this one in and we'll have a look at a couple of different elements to netcdf s so first thing it's going to ask me for is variable again the tool helped really useful here and it will ask you what variable do you want to view so I know from emailing with Derek that this is solar radiation I think without cloud on solar radiation with cloud or something along those lines it's not hugely important because we are just looking at how model builder works but we've got two different variables and at Laskey for the x dimension the Y dimension and then we could actually add in some band dimensions so if this was say the solar radiation with cloud and without clouds every 15 minutes we might have a band dimension of time and we do have that so if we put that in there we'll make a raster out of this netcdf layer so that's okay that good so we have our netcdf layer here and this is the solar radiation without cloud I think and if we open up this layer and have a look inside it on our layer properties we've got this new tab called netcdf we can change our variable so I'm going to make it the one with C which I presume is cloud and then if I apply that should say in the background how the raster changes the band dimension currently is set to time but if we go into symbology I don't appear to have any different bands there's only one band there that's a bit strange let's put it on this blank one and here we've got the dimension of time and it looks like it is just a snapshot so it's the first of January 1985 at midnight okay so these netcdf it's strange we could probably have more things in them so less files to begin with but if we look here in the ones that I've created previously that we have 1985 1st of January 2nd of January 3rd of January so it looks like it's just going on a daily basis interesting to know we know a little bit more about our data now so we can begin building a model so I've created a new model if you need to know how to do this create toolbox create model watch the rest of the series we have in-depth stuff on that so this is my new model it's in my tool set toolbox which is called netcdf clip and we're starting with a blank model it's a good place to start and what I'm going to do is open up the multi-dimensional tools and I'm going to take make netcdf raster layer and just drag that across into here so we can see how this tool looks inside model builder now what we'd like to do what Derek requested is that we can iterate through is 10,000 netcdf files and the aim of this is to be able to cycle through those create a raster layer for each of the different variables so with cloud and without cloud and then finally to clip the output by the X of Ghana so I've taken this Ghana shapefile from a world shapefile that have on my hard drive and you can find country shape files wherever so we do know that we are going to need to create one raster for each variable now there's several different ways to do this mine might not be the best way but hey we're just going to do it this way so I've got the netcdf layer the raster layer and it's going to output a raster layer now what we would like to do is to be able to iterate through on netcdf files so we do the same action on each netcdf file that the model builder comes across so I'm going to go into the model or rather inserts and I'm going to look for iterators because that's what we want to do we want to hit each file now ArcGIS doesn't have a preset for netcdf files we've got workspaces tables rafters blah blah so I'm going to use just files and here we can see that we've got the iterator in there that's good and what we want to do is the file that gets iterated we want to use that as the input for the netcdf raster layer so I'm going to join these up with the magic wand very nice now I'd also like to tell RT is just to use netcdf files so I'm going to go for file rename this and I'm just going to put that dot NC on the end ok now where are we iterating from that is a good question and we are going to iterate from folder because we have the folder in our set up here which is called files and so I'd like the iterator to look in the files and I like it to iterate through there you can see when I go in to add data but it doesn't recognize the dot NC files but every model builder will so I'm going to go back to model builder and I'm going to insert a variable and that variable is going to be a folder there we are so the folders end I'm going to connect that up to my iterator and now the iterator will know where to look so at the moment all we're doing is making a netcdf layer and we're going to put the output raster layer so if I tell the model builder where this is this is kind of like hard coding the location so I'm going to go into the here and I'm going to set the folder as this files folder okay that there we go that's set and you can see that the elements of the model become colored and that means that you know things are set right when we get to the netcdf layer it's still white black and white and that's because we don't yet know what the settings are for this netcdf raster layer so let's make that as well we need to know what variables so for this one I'm not going to use name I'm going to tell it to use one of these files so I'm netcdf files we'll just use the first one and that will tell us our variables they're all set now that's great this is long flat the output raster layer not bothered about what that's going to be called at this stage that's all good so if we okay that you can see that we have an extra input here and I'm actually going to delete that simply because if we go back in here you can see that our variables are all still set which is quite useful and instead of having a single file I'm going to attach our iterator back to the model and there we go so we found out what fields we've got available we've hard-coded those into this particular bottle tool and then we've got our output so let's give it a test and see what happens that was apparently successful let's close that down and oh we haven't added this to display so let's make sure this gets added to the display now it is now this one I know from the data is actually just the last netcdf file so that's the only one that we've got showing and the reason for that is that we don't have an output location set so all ArcGIS is doing at the moment is visualizing this raster from the netcdf file so you can see here we've got our location showing where it is and you can see that it's the last netcdf file that art came across and it just pings that one up so in order to get all three of them what we could do is we could set an output location so I'm going to insert a variable and that is going to be a workspace so if we go down here got workspace okay that and I'm going to set this workspace to be a file gdb and I'm just going to call this demo that okay now we don't actually need to connect any of this it's not necessary at the moment okay that so we have our output gdb we have the iterator going through our netcdf file at the moment for the non cloudy areas or the non cloudy variable and the final thing that we'd like to do is to clip our netcdf by the extent of the country Ghana so I'm going to go into my date management tools into rosters and raster processing and from there I'm going to steal the clip and put it in here what we'd like to do is to clip this by Garner and Garner x' up here in my table of contents if a double click on the clip the output extent is going to be the same as Ghana there we go the output raster data set at the moment is going to the netcdf clip and I'm going to change that to the demo and you'll also see that the file name is going to be sis underscore layer 5 underscore clip so okay that I'm just going to tidy things up here a bit so we've set our demo gdb as the output workspace we've set the name to be s is layer 5 and just go clip we've got garner is the clip function and I'm just going to make sure that this gets added to the display and I'm going to stop this one from being added to the display so let's remove that and then I'm going to run it again now can you foresee a problem with this particularly with the file name we're iterating through a number of files which is fine comes through here it gets clipped by garner for the output file it's always going to be the same and that could cause a problem so let's just give it a run and see what happens I'll click the validate first and I'll give it a run and it's going to do it three times and it's done there we go now we've got three nuts EDF files and it has failed to clip for us why is that use input features for clipping geometry I forgot to check that there it is okay so that's okay that let's validate our model again and then this time I'm going to run it but I'm going to let you watch what happens in the background let's just make this a little bit smaller cut that in there and let's go though it is clipping them all and fantastic so we've got our layer 5 output click we know that it ran three times let's go and have a look inside our demo gdb wait there's only one why is this happen dark what are you doing to us the reason this has happened is that it's got the same file name so what arc is going to do although it processes all three netcdf when it outputs them it's got the same file name every time which means it just overwrite so we need to change that can be quite tricky to do when you start with modelbuilder but once you know what you're doing it's absolutely fine so let's cancel out this and look at how we can get individual names for each netcdf now in order to get individual names for each netcdf that we're creating we could do with something that is unique and fortunately if we look our original netcdf files and look like each one of those as a unique name for every day and also fortunately arc reads this so if we go back to the start of our model write our folder is that the files are being read from we are going to iterate over those and one of the outputs from that is name and we can actually utilize that name to give the name of our files so if we go to our clip in here and we look at the output raster data set at the moment the name of it is SS sis underscore layer five underscore clip now we could get rid of this layer five part and if we use the percentage sign that means that we're going to be using a variable so we're going to give our K variable name and the variable name that we want to use is also called name so if we do that and we close out the use of a variable using our percentage sign as well very good if we okay that now you can see how the output name has changed and it's going to be whatever this value of name is on each run of the model so let's give it another run and let's see what our outputs are like this is looking far far better Hey look at this so now we have the name of the original netcdf file and we have the suffix of clip and we have the prefix of sis very good so that's looking alright we've still got a letter v in here because that's from the last run of the model so we can just remove that that's fine so this is looking excellent actually we're almost there what else we'd like to do at the moment on that CDF is only looking at this variable S is and what we actually want is for this to be s is C as well cloudy variable too so if we okay that we can get around doing a lot of stuff again just by doing this and we can put I don't want the demo gdb to be included I'm just going to copy all of this and bang paste it in so now we have got a duplicate of all these processes and what I'm going to do is click into this one and instead of sis we are going to use sis C okay and then on our clip instead of sis underscore name I'm going to make it s is C underscore name okay that so now we have two outputs we've got Ghana being used twice we could probably attach that one to there as well let's just validate this model I'm going to get rid of this goner just to make this a little bit tidier there we are and that should be alright let's have a look Ghana want to use the input features clipping geometry or good okay so now what we're expecting is that six files will be output one for F is 1 4 s is C will all be clipped to Ghana and to see this in action I am going to remove all of these and then we're going to run the model again if it goes file one oh that is interesting so it looks like for our sis see we are getting around the strange output hmm classified it for some reason we can change that quite easily by going into the symbology and making it stretched okay that now that looks the same these sis C files do look rather different to these ones I'm not sure why that is perhaps Derek can get in touch by the comments and let us know exactly what this data while these data are showing but there we go and so that's how we can iterate through netcdf and we can build not a complicated model quite a simple one there's one other there's quite a lot else that we could do with this to make it bad but I've been blabbering on for ages so I'm going to leave it there for now and if you do have any questions if you want anything clarifying if you've got any suggestions how to make the bottle better get in touch just like Derrick did and I will do my best to help you out so there we go that's netcdf how to iterate through them using model builder thanks for listening give us a thumbs up like a really important it's great to see so many subscribers as well so more of those please if you haven't subscribed already and thanks a lot for watching happy mapping

Channel: burdGIS

Views: 9,208

Rating: 4.9285712 out of 5

Keywords: burdGIS, GIS, mapping, geospatial, cartography, netCDF, iterate netCDF, modelbuilder, #modelmayhem, ArcGIS, Arc catalog, ArcMap, ArcGIS desktop, modelbuilder netcdf

Id: cxzFlyXLhP0

Length: 23min 2sec (1382 seconds)

Published: Mon Aug 07 2017