GIS Python Tutorial: Getting Started with ArcPy and VS Code #GIS #Python #Code #ArcPy

all right so welcome to class today we are going to do a lesson on um hands-on using python using an ide so what our goal for today is i'm going to show you um if you've never seen it before a tool called vs code raise your hand if you've used vs code before if you're familiar with it well most of you have which is great okay it's a free and open source ide integrated development environment and we're going to use it to start writing some python code that we will then run inside of arcgis pro and then next week i'm going to show you how you can hook up vs code directly to work with the arcgis pro python libraries without having to have arcgis pro open so our goal for today though is just to introduce vs code put some code into it and um do some sort of data processing and i've basically i've given you the code that we'll work with and the code we're going to work with is very close to your lab for this week which is about um csv parsing okay we'll go over that so to start with again because i'm having a technical issue with um excuse me with my courses go you should have on under the week four um week four on my courses there should be a file week four datasets underscore two you see that there so that's different from the one on tuesday so go ahead download that week data sets underscore 2 to a spot where you can easily find them and ideally write files to so in my case as i always do mine is in c temp so if you're using one of the computers in the lab here you can create a c temp folder if you're using your own laptop it doesn't really matter just a spot that you can remember to get back to and after four weeks now you can see this is sort of my sort of approach i use in every class okay so once you download that zip um when you right click on it and do you know i'll use 7-zip i'll extract it out to this folder here then inside that folder is another folder um and you should see four files and this is what we're going to use for um for class today so i'll give you guys just a second to uh to to do that so you should have a csv file a ppt file a py file and a prj file if you know what all those all mean that's that's awesome but we're gonna go over them all so does everybody have that so again download it unzip it and that's what you should have in your uh you should have okay okay i'll give everyone just a second to kind of get there so we don't uh leave people behind okay and while we're waiting i'm going to go ahead and open up the pptx yup what's up no it's in the lectures week 4 lecture yep yeah i'm sorry that i don't have my courses working just to visually show you let's start with the pptx this is just a high level sort of visual summary of sort of what the mission is here that what we're going to do in class today so what we're going to do is take a csv file and that's the um the harvey 311 data set that was that you downloaded and we're going to take and read that file in using a combination of the the csv libraries of python inside of visual studio code in arcpie and then we're going to with that code we're going to generate a shape file that's going to pull the contents the contents of the shape file are going to be drawn from the csv and we're going to display it inside of arcgis pro so that's that's the goal of this that's the algorithm as it were and so forth um and so let me just give you a quick demo of what the whole the process will look like um so that's that's the the visual any questions on that by the way is that clear and what the what we're doing for the most part have you guys has anybody ever like use python to access the contents of a csv files anybody yeah a lot of commonly yeah okay great so if you've taken the standard intro to for cs you're familiar with this now in general even in the gis world csv files are very common and if you take a class like an intro to gis class there's actually tools inside of arcgis pro that are designed to like read csv files directly in and render them as points because this class is about coding and algorithms we're going to do it with with python code okay so let me just show you the the idea here i'll close that um no maybe i'll leave it open so i'm going to open up arcgis pro and i'll go ahead and create a new blank map and i'm going to save it in my c temp folder i will not create a new project or new folder and you don't have to necessarily do i'm just you can get arcmap open or arcgis pro open you don't necessarily have to follow what i'm going to do right now i just want to show you what's going to happen here so if i go to view python window and i go ahead and i right click and i do load code and i go grab that code i gave you for class today which is in the week 4 data sets it's called week four prep dot py open that up it's going to do this code is going to do um oh wrong one it's going to do this workflow the code is going to read in a csv file use arcpi and python to create a shape file define the fields in the shape file and then populate the shape file with data records from the csv all right so what is that going to look like here here is the code we're going to go through it all in just a moment um but let me just run it to show you again i just want to show you like the big picture of things and then we'll go into the details of how it works and so it has there's only about i don't know about 15 records but there it's done it ran so this is data for i should have mentioned this this is data related to 3-1-1 calls you guys know what a 3-1-1 call is never heard of three one how about nine one one hopefully you know what nine one it is so three one one um is is a is not is a non-emergency phone call and so these are data records from hurricane harvey from texas from 2017 and so if i make these a little bigger even um so that my code created the shapefile if i look behind the scenes um i had to go to a directory called see uh temp csv and there there is the shapefile and um because the shape file is currently open in arcgis pro that's why you're seeing these lock files right and then we didn't really look at the underlying data yet but if you if i open this up and you have this file i gave it to you 311 harvey this is what the shapefile got created from right so there's about 16 records in this csv file and i'm just opening it with excel just to look at it and i'll zoom in a little closer right so if i do that concurrently with opening the attribute table so again this this this that what we're looking at here i just created it using that code i gave you and it's got um basically three columns that we created and so i i with my code that we're gonna go over just a moment i only grabbed certain this csv file has oh i don't know um 26 27 28 has 29 columns of data right but i don't i didn't want all 29 columns in my shapefile the ones i was interested in programmatically were the latitude and the longitude does anybody know what the these um these things are they are they are coordinates anybody know what particular kind of coordinates there as opposed to like the x and y that's that's getting close that's a half that's like a half answer but good that's good no there's a real specific term i'm looking for anybody no these are what are called decimal degrees so you've probably seen them when you use like google maps or something but this is a latitude and longitude coordinate that's basically expressed i guess you would call it a double you know we're using programming terms it's it's a decimal point number as opposed to like 29 slash north you know 86 minutes 27 seconds right that particular wave expressing coordinates doesn't work isn't it just isn't easy to work with um and then super bonus points does anybody know what this is but these are i mean they're x and y right so they're on a cartesian grid any of you taken the 382 class that's before this class that's close no that's very close it's it's not what they are but you're in the right direction what was the other coordinate system you learned about there were three of them in that class exactly this is a state plane coordinate system for texas specifically so if you get if you take a class more on mapping concepts there's different ways that things are referenced on the surface of the earth latitude and longitude is the most commonly used one and from a programic programmatic perspective programming perspective decimal degrees work good but sometimes you'll find data especially from the united states where they'll have the state plane coordinates and they use those um and you know you might not know that it says x and y so that should clue you into something but so that's this is the data that i created the shapefile with and like i said it's outside the scope of the lesson today there are tools in arcgis pro that's that are designed to read what they call xy data where you can take a csv bring it in and it'll sort of find if you if the data has coordinate values as in the records it will read those in as points and just automatically generate a feature class um which in many cases that's fine but for what we're learning about in this class we want to use python to do this because i mean ultimately your ability to use code gives you way more control over your data and if it's part of some bigger process the out of the box tools really won't help you per se so what we're going to do if you look so i'm just going to highlight this column i'll make that one yellow and i'll make this one yellow okay so if you look at sr type overdue and then the latitude and longitude which is off the screen those are what the code i gave you those are the um if you look over here now those are the uh the values that i brought over from the csv and populated into the attribute table of the shapefile when the code ran okay and i also brought the overdue and i also um i have it if you've already been looking at the code i had to just generate an incremental number to create like a record id and there is a column that's automatically general called ib but i just created my own column called record id okay so again i pulled three of the sort of 28 columns i only pulled values out of four of them and they use the latitude and longitude column the values in them to create the shape the shape geometry okay um and so any questions so far and again i'm just trying to explain what we're going to what what i'm going to show you how to do pretty clear so far on on the process again i don't know if you guys have done a lot of data processing that's such a big part of gis world so that's why i tend to do these as examples now another thing um just to go over make sure we're clear if some of you are new to gis in general let's talk a little just more about attribute tables just by remember we talked about like different data types the concept what's an example of a data type how about what's this right the guy right behind you is on his phone there what's the type what's a data type there were examples of data types one of them was a string right data types and programming so what's another example of another kind of data type what's that yeah like integers doubles and so forth right so when you when attribute tables and their fields in those attribute tables have a data type associated with them so if you were to click on the attribute table if you have one open or if you look at mine when i right click on it and go to fields you can see that every column has a data type associated with it right and if you were to add something new you know when you create it manually you give it a name a data type and some other things the reason i want just to mention that when we when we go into the actual code part of it is when you're creating those fields programmatically you have to set the data type okay and i just want to make sure you guys were clear on that and the general idea um so it makes more it makes sense when uh we do it with code okay okay so there any questions ends on what we're gonna do no all right let's go ahead and do it so if you if you have never used it or want to get your own copy we're going to use vs code here's a link to it it's a free and open source code editor and i learned about it from students years ago um and so there it is just again if you want to get your own copy of it you don't have it now on most machines if you have it installed you can just go down to the search you can see it's here because i used it recently but you can just do the old vs and then look for visual studio code my understanding it's like a lighter weight version of visual studio which is a much more heavy duty robust professional you know you're gonna lay for deaf level kind of software um but i'm gonna go ahead and open up visual studio code okay now one of the first things you should do if it's not already there visual studio code has the idea of extensions which are little bits of software that you can pull into visual studio code and python is actually one of the most popular extensions available and so to get at the python extension if i click on this um this icon here it's kind of little building blocks being added um mine is already installed do you guys have python intellisense pylance is that showing up right at the top yeah so go ahead and um you know i'm not gonna go ahead and install that and that'll get you started with um with python now another thing to mention if you're watching this video and you're trying to do this on your own computer you will need to have a copy of python on your machine and if you don't have that if you look down here under the quick start install supported version of python on your system you'll have to do that you'll have to then point the python extension at um at your version of python on your system now from previous experience with the with the house computers there should there should be there's already a version of python that's sufficient for our needs okay so does everybody have does everybody have the python extension can we move forward just make sure no one's left behind here we're good okay so if it's installed we're going to go ahead now then go file new file and then right out of the gate i'm going to do a file save as i'm going to save it in my c temp directory i'm going to call it um class 4b for this week and then under save as type i'm going to go find python okay and that'll tell visual studio code that we're going to be doing python coding and then all the things sort of associated with that will start happening so hit save okay now a really important thing to note by doing that by creating a python file take a look down here on your bottom left okay you should have what's called some whatever version of the python interpreter should be showing up so in mind it's python 3.9.7 64-bit what is it what do you guys have out there yeah okay you have something though right i mean if you if does anybody have a word saying select interpreter anybody seeing that no okay again if you're watching this video later and you're seeing it saying select an interpreter like you're working on your own laptop that means you have to go and get python on your machine and then point it at you know from here you have to point it at wherever it is so if you look at mine um uh let's see yeah if you if you look at the i don't think how well you guys can see it on the screen if you look on the bottom left it's like c users brian t app data local flat flower you know my sort of my instance of it and what we'll talk about next week is how you can um you can clone your python environments anybody familiar with that idea i know some of you guys have done a lot of this stuff but cloning your python environment you familiar with that concept is that in the cs intro no okay so just to foreshadow what's coming next week so you basically have sort of different uh how would you just how would i describe it like environment so an environment is a collection of all of the different software libraries associated with a given version of python but you can often clone that so like i i don't want to use 3.9 with certain ones i want to use a clone of it with some extra stuff and i want to mess my other one up and we're going to see in arcgis pro how we can actually clone the python environment that comes with arcgis pro and then point visual studio at that and therefore we can start to run programs without having to have arcgis pro open but for today our goal is just to get arcgi uh to get visual studio code open get some files going and so forth and practice so that was a long lined up for just make sure you have something selected there okay now typically i don't like to just give you guys the code um and copy and paste but we'll do it but i i did um because of the my courses issue and i i didn't honestly have a better way to do it so what we'll do now is we're going to start to slowly migrate in our um [Music] code now if i go over to the code that i gave you if i go back to that week four data set folder we're gonna start working with week four prep py and i should mention if you guys are bored uh if if my lecturing is not getting your attention i will mention that you're if you want to look at lab 2 lab 2 is very heavily based on this code file so if you want to start trying to figure out lab 2 um you're welcome to do so i always want to have you guys feel engaged and so forth now you know you can go ahead and just kind of open it right up um in visual studio code and one way you could do that would be to just drag and drop it in there so i'm going to drag and see how there's like a little kind of gray up in the left all right but just for okay so yes you could just copy and paste it all into your thing and be good to go of course that's not really the best way to learn it so let's just practice just for the practice of using visual studio code and again i know a lot of you already used it but let's go back over to our blank one we'll use this as a reference and so the very first thing we want to do is put a comment in so i'm going to do um pound character okay and i think it was control nope okay can you guys in the back see that okay i try to make the screen a little bigger okay and so if i look back over to the this file i'm going to import arc pi import sys and let's actually start just let's just type an import statement so again i am okay so intellisense it's already starting to fill in for me so just by typing the letters i am and i hit enter it starts to fill in for me right and it starts to tell me things are um you know it's looking for something so again as a very introduction to just using vs code or really kind of really any ide this you're going to see this kind of thing everywhere um we're going to import arc pi sys and csv in fact let's do csv first c s b okay so csv is a native sort of library that's part of this python interpreter right so we don't have to do anything special we don't have to like download it install it and that's kind of what we'll do next week i'm going to show you about how you can pull um libraries that might not be in part of your python environment into your python environment and use them now just to save time um we did csv i'm just gonna i'll just highlight let's copy those ones in okay now let's take a look here just just to note so this is also kind of related to what i was talking about for next week the arc pi python library is currently not inside of this particular um inside of this particular python environment hence you can put the import statement for it but it's the interpreter is not finding it in there it's it's yellow so it's more of a warning or we can keep going but it's but next week we'll we'll fix that where arc pie will show up and so forth so save our work and let's move on to the next part of it so because we actually don't have arc pi uh things won't fill in so we'll just copy line seven and eight okay now now arcgis pro um has something called environmental variables and so that's what's happening here with this line so let me see if i can find where they are um yeah so you see here under the analysis tab environments this kind of thing is way outside of um what we're doing in this class but it's more just make you aware that arcgis pro as a software has sort of global settings for the software the environment so for example the current workspace this is where your you've learned about file geodatabases this is where things are going to be sent to or where things are going to temporarily be put like the scratch workspace right and on and on there's a whole bunch of these things the size of when you're doing analysis with rasters and so forth so basically what that line of code did got at one of those environmental variables about overwriting outputs which means when the code is running and we'll see that because we'll run it a couple times it's going to just overwrite whatever whatever it's it encounters versus like you know how when classic you know you save a file you want to overwrite it do you want to really overwrite this it'll skip all that and won't give you an error so that's all that line does there but it can be handy when you start to run multiple versions of the uh you know you've got to fix something and so forth okay all right let's do this now the root directory let's practice just typing a string in just for that so so rot underscore okay and i'm you know and then i would add make this match your system okay now you guys know i like to use c temp so i'm going to type in okay anybody let's just do a quick refresh um anybody know remind me again why i put two slashes in yeah right someone tell them someone else tell me what an escape character is how about i'll go over here i'm just trying to mix it up but i appreciate your input what's an escape character okay can somebody else give me a way to explain it to like if i was a non-computer person right yeah exactly those are all great answers um escape character is for sometimes in encoding languages right there's certain characters you can't put inside of a string but you can with the escape character so it's sort of like a switch being flipped to say let us use um that so in this case c temp is going to where a lot of things are going to happen so look let's say for this week i want to later on in the code if you've already looked ahead to it we're going to be accessing the csv we're going to access a prj file so maybe i want to make everything happen in that directory so the way i would do it ctrl c i'm going to copy that out of file explorer put it all in there and then just go and you can almost kind of see actually um if you didn't hear it uh i i don't even know what escape t is that a tab is that what that would do tab character but you can see how notice how the letter c is orange colon is orange then we get a yellow slash and a t that would be the escape character for a tab right so if the string wanted to print it would be like c colon with a big tab blank space so go ahead and make all of your the whole directory path um and you can see now even by doing that it's it's telling you to kind of keep the um and then put two slashes on the end of it okay okay now sometimes you want to give people some feedback about what's happening in your program so that's all that's going to happen here this is just telling you that the we're going to create a feature class okay okay now a thing with shape files is they have to have what's called the projection file and if we look um at the files i gave you for today that's i gave you this wgs84 if you're curious about this stuff if you right clicked on and just did like an edit with notepad plus plus um this is all the sort of plumbing behind the coordinate representation and projection of the shapefile and again if you take this isn't you don't this is anything you need to really worry about if you take some of the other classes this kind of thing becomes more important the re the way the reason it's important to us though is we need to use it when we're programmatically creating a shapefile right so there it is it's just a text file with some kind of um you know anybody i don't know if you guys know what a spheroid is but this is like i think the field is called geodesy the measurement how you actually measure the earth you know like surveyors do this and kind of stuff and like you know um datums and um spheroids which are representations of how the shape of the earth is and so forth so i'm gonna go off on a tangent so like most most uh if i understand correctly most phones the gps on the phone uses the wgs 84 sort of datum that's been the world standard and so like the same coordinate can have a can be in a completely different spot based on the spheroid representation of it and again if you take other classes in this on this topic you'll appreciate that more but for what we want to do we just need to put it in there because when we go ahead and create when we go ahead and create the shape file it's going to have a prj right so we're defining the projection of that shapefile in the code and so what we're going to do here is um that move it over yeah nope i'll just copy this one okay so again make sure that now we're going to plug together root directory and then i've hard coded the name of the file but if i look at that root directory right that should all add up to a valid directory path and then wgs 84 should be in there so if you haven't already done it um i'm going to go again back to the files i gave you or do i already have it in there we need four datasets yeah yeah it's already in there in my case okay c tab week four data sets yeah i have it twice i mean yeah i could clean this up but if anything it's a good practice i've seen a lot of um students in the class kind of get tripped up in this particular part of these codes i've been giving you of just knowing the directory so in this case c tab week four week four data sets with the two slashes on the end then we stick that together with the string wgs84 it basically just puts this all into a string variable called prj that you'll see why we do that in just a second okay all right so this is some setup so let's go ahead now and um i'm still recording let's go ahead and do the next part of it and those we'll actually let's just do this we'll we'll put this next these next lines in and then we'll run we'll run it and just see what happens okay so we're going to create the shape file now using arc pi now even though we don't have arc pi in the python environment we're going to run this code inside of arcgis pro and as we heard beginning of class we don't even really need to use the import but it's good practice to have it there so i'm going to copy those lines over and let's just go through what they do here's here's the documentation on it and so it can be helpful to to get used to looking at this kind of stuff if you're going to be coding with these things but basically creating a feature class um is one of these input output kind of functions of arc pi where you have certain parameters that it needs to um know like where is it you know where the feature class location where is it going to go what's its name what's its geometry um the template uh which we're not going to do that that's like if you have uh you know do you guys know what templates are the general idea of a template it's like microsoft word you're gonna start you have things already set ready to go right um [Music] m values i don't even really remember what those are i wanna say linear measurements um i think i think it's directionality if somebody watches this video later they can correct me z is if it's sort of three-dimensional coordinate system it is optional but we're going to put it in there this is where we plug it into um the prj file and we're not going to worry about all these other things but if you look i mean a lot of these lessons i do are based on i look at the example code right and i think the syntax has changed a little bit but it still works do you guys remember syntax you know syntactically when you see the curly braces what that meant with some of these functions yeah yeah exactly they're optional and you might remember when we did buffer um and you think your lab one you had to do some of the optional things right curly braces tell you they're optional so what does this all look like you know i'm not using all of it when i go back over to vs code you know i'm telling you where where it's going to go the root directory what i want to call it what type of geometry the template the z and m values are disabled and then i am going to give it a spatial reference um by passing it in a path to the p that prj file right so that's how i'm plugging all those parameters in to that function okay and i think at this point we can run it and just see what happens so let's go ahead and save it let's go back over to arcgis pro i'll turn that one off and where's that python window i'm going to go back to the python window and by the way just as a re refresher you can right click in that white area and do clear transcript and um the method we're going to use is we're going to right click on the cursor area right click and do load code and then we're going to go browse to where that that file was that we had in vs code and where was my code where did i save that thing so that's whenever you forget i do a save as i saved it in c temp is where i saved my code file okay we're having this issue okay now if you have this issue that i'm having where i save that file and see temp and it's not showing up or anybody else experiencing that all right the way that i found to fix that which is really annoying um i can't do a refresh here i'm going to go to catalog then i'm going to refresh there hit i'm so i'm going to i'm going to go to folders wherever find the folder that you saved your file in from vs code right click on that and do a refresh and some things showed up then go back to the python window do load code go to c temp and there it is why you have to do it that way i don't know but it works i don't you know just moving forward here right so that did you guys do that did it magically show up now so there's something about the loading code with the catalog it just needs a refresh but so hit okay and we'll see the code that we wrote in vs code is now showing up over here and if this works correctly all it should do is just create in that in that directory it should create a shapefile with basically nothing in it a blank shape file so let's hit enter boom and it kind of happened pretty quickly if you see i'll take the old one out just so it doesn't get uh so what do i have if i look at that that just got created if i go to that directory the week four data sets right look at the time stamp um 11 41 is that what time about what is right yeah it created it it added it to the map it put the locks on it and so far so good really it's good now if i look at it though if i right clicked and looked at its attribute table there's nothing in it it's just blank because all the only it got to like running this line creating it but that doesn't have anything in it right now i could go manually and you know add fields to it i'm sorry the python window is acting weird um i can add fields to it and do this whole thing right if i wanted to manually but this is a coding class we're going to do it in code that i'll show you in just a minute um it's over did you guys get that to work okay again even we're in beginnings of this even just loading the code and i just want to make sure the whole workflow is happening for you so um let's go back to vs code then if everything's working worked okay for you okay so now i'm going to grab um things related to the csv file so go ahead and copy line 24 through 31. okay so again the way i've structured it everything's driven by the root directory so all i'm doing here in line 24 is i'm just setting up a string to point it at the csv file right that we looked at over here and it's just again csv file is just a text file right um and um you know if i were to go to see even though it has the excel icon if i were to right click and do open in notepad plus plus uh there it is right it's just a big text file with commas and just to be safe too i'm going to close this okay now actually i probably should kept it open huh what i'm doing here later on in the code if you go back to this graphic you know i'm taking the content selected contents out from the csv and pulling them and putting them into into the shape file so the way that i typically do that me personally i like to just define the column name so we're going to go after four columns i had them highlighted in yellow and i closed it but um we're going to go after latitude and longitude um what else did i say overdue and what else is the sr sr type here it is okay so at least in my experiences like i said when i when i've done data processing and i know that i'm going to be going after specific fields so eventually you're going to see in just a moment the code where we're going to go into this file and we're going to give me whatever is in this column the sr type column and then give me whatever's in the overdue column and give me what's in latitude longitude so you have to reference those field names the way that i do it typically is i do it like this okay so what i'm doing here is i'm defining strings that are equal to the names of the fields that will be accessed and can you anybody why do you think i do it that way if i've given you enough setup for this yet oh what's hard oh yeah go ahead when i say like those of you done coding um you've probably gotten the philosophy of trying not to hard code things so what is what is hard coding what is that general idea i've hard-coded this yeah oh that's an interesting answer i didn't think of that's okay uh okay that wasn't the way i was thinking of it but um anybody else have a thought on the idea of hard coding something but yeah the context of control you're writing you know if you have it set to one thing so you can make it that one thing like you have again right but it could be five six seven three four but you don't want that to be changed right yeah i mean basically the idea of hard coding is what you've committed you've committed some of the code to being a certain thing with like a literal string value and if that thing changes later it can cause problems okay so this might not make a lot of sense right now but the idea i did do it this way let's say this is our this was data coming from the city of houston texas right and one day they decided for whatever reason they're no longer going to call it the latitude field they're going to call it the lat field right and i've just spent hundreds of hours writing code that references the latitude field right so instead of hard coding my code to say the literal string latitude i put it into another thing that i can reference kind of universally right um so that's that's why i do it that way it gives me more flexibility i don't want to hard code things all over the place because if i start using if i start referencing that field in multiple places which i'm not sure if i actually do in this particular code but other code does if i have to if i have like 20 instances of using the literal string latitude that could be a real nightmare to have to update it later so i like to try to always put these things kind of in the beginning what they are and then if later they change it i just change it in one spot and it kind of propagates down okay um so it's just a way of thinking about your code again it'll make more sense in just a second um why i do it that way so let's move along so those are basically the four fields i want out of the csv and now similar thing for the shape file you just saw we created the shapefile but it had nothing in it it had no it had no attribute fields and anything and that's this next part we're going to kind of define those things okay so i just copied lines 34 through 42. and you might remember i think i actually took it out of there uh if we look back at this one this is this is an older version of it if i look at that attribute table remember i went over that when we started it's got record id overdue sr type right so this is the code now that does starts that part so similar thing i don't want to hard code the names of my fields i might want to change them later for whatever reason i don't want to rely on the string record id i'm just i'm even using like this is at least how i do it i call it field name because like it's almost like documentation right um it's precision and and so forth um the the overdue field which is going to be a double we'll see that in a moment in the sr type which is a text and all of that stuff there relates to what you see when you go over to um over to fields right um the name of the field the format uh precision you know there's that nine from record id you know again you can manually do that you can type that all in but this is how you set that stuff up in the code right so that's what that does um [Music] so we're getting those fields ready and now let's take a look at um let's run this part let's bring this over we'll run it here's now where sort of the magic happens of adding the actual fields in to the shape file so i'm going to copy lines 45 through 48 okay and i put the reference for the code you know where you can find this in the in the comment if you want to look it up but this is you know this is how you know programmatically will add a field to a feature class using our the arc pi library and um the table the field name the field type the precision those are all things we kind of set up to not be hard-coded okay so if i go back over here and just kind of look at some of these um you know so there's arc pi the ad field management and then the parameters you know where is it going what's the feature class it's going to go to that we created in back in line 20 which is still in memory at this point um then i start to plug in my stuff from up here that i didn't want to hard code in in case i change my mind later so record id comes there um the data type which is uh i mean that is a hardcoded string the precision the alias nullable all the things that you can do to create um fields in a feature class okay so let's run that let me check my time here too i'm doing pretty good yep so copy that in there's about 45 lines of code now with you know gaps and stuff i'm going to go back over to um what happened arcgis pro so here's right here um let's close that let's close that let's run it again i'll get rid of that old one so again the python window clear transcript going to right click in the cursor area do load code and go to um c temp and just you know i saved it yeah 11 38 okay so now there's my extra code in there now when i run it again not much really it happens kind of fast nothing really visually exciting happens but what's going to be different now compared to what we had before and by the way i overwrote my old one because remember we did this thing back on the round line what about line six or seven i overwrote the output so i just overwrote my old one from before now my new output is here when i open the attribute table there's still nothing in it but i've now got my three columns right which compared to what we did about 10 minutes ago all i did was just create the shapefile with nothing in it now with the code using my code i've got three columns that have got created because of what of this part down here do you guys get that to work yeah okay i mean it's easy just copy yeah okay if you're getting that error input spatial reference double check all this make sure that directory path can find that file what's that okay how about let me um let me finish up let me let me do the next sort of part of the lesson and then we can come troubleshoot things okay all right because then the next part is this this whole chunk of code here that'll take a bit to kind of go through here um but i put i made a point i really updated it with comments to try to really step you through all of it all right let's just grab this big jungle code here so from line 50 just go all the way to the end line 50 to 79. copy that because this is the real sort of meat of the whole thing here as we're or plant-based protein if you're vegan yeah um okay so let's walk you through this so what's basically going to happen here is we're going to open the csv file up transform the csv csv file content into a dictionary which is a key value pair do you guys all know what i mean by key value pairs key values so the idea of key value pairs um think of like if we remember what whoops do you remember uh what a list is let's back up what's a list in python that's a real basic concept yeah a collection of items right um so a key value pair imagine that every item in that list has sort of like like a key or a reference and then the value you know is the actual thing right so that's what dictionaries um for those of you that have code in other languages i want to say like a hash is that what a hash map is when you have a key value pair yeah so if you want you know i put the reference to the um to the diction the dictionary reader if we want to look at just look at the documentation real quick what you should do creates an object that like a regular but maps the information approach to a dick dictionary whose key so if you didn't know what a key value or dictionary is click on this link and so dictionaries kind of like if we look at this example here from the python documentation i'll just zoom it in little too far zoomed in so if you look at it it's basically like a list right each element here this would be sort of one element in the dictionary and then it gets separated with a comma but in this case there's a key value in this case this example here is jack colon and then the number or value associated with the the key right so like you know actually today's my birthday so mine would be like you know your birthday brian colin 9 16 um sally colin 9 17 whatever the key and the value pairs uh so that's what a dictionary is it'll it'll it's going to take all that csv content and use the column as the key and then everything that comes along will have um will be the value associated with that it's actually a really really handy really incredibly handy method to transform a csv file into a dictionary because when we when i walk you through the other code you'll see how we can then just grab the content right away and next week when we do more we're not doing any like debugging or like break points we're basically using vs code as a glorified text editor but we really run code out of vs code directly you can really inspect what's going on inside of variables and it will kind of make more sense okay so we basically take the csv object the dictionary reader sort of method pass in the csv file and it becomes the variable called current data okay then um we start this is the method i use we're just going to create a basically an integer variable called row id val and sort of i'm calling it seating it to zero right because in just another line it's going to start looping through all the contents of the csv file you know each row is going to get hit okay and so that's what happens here then a for loop right and you guys now even if you're new to coding you've seen for loops at your lab one had a for loop and so forth and so it's going to go through now every row inside of the csv that's become a dictionary and if we look at this line here's row latfield so what do you think's what do you what do you think's happening with within that particular bit of code right there how about somebody we haven't um i think brianna brianna what do you think's happened in the part i just highlighted on line 56 what's happening there you think i can't hear you i'm sorry what yeah so do you reckon anybody recognize what this is right remember what i just said about five minutes ago why we did all that what is it someone's remind us what it is how about the i don't know your name no the guy behind you there sorry what's your name chris what's what what was that we just talked about and it sort of comes in over here or maybe a sister or brother remember okay yeah so you might remember back up here you know lat field long field we define the names of those fields we want to get at from the csv and that's why we did it so as opposed to putting you know i wouldn't want to do um something like this i mean i could you could do it you could do that but that's that's hard coding it because later on just even looking down a couple you skim ahead to line 64 there's long field there long field there so if i had this literal string in multiple places it could become really chaotic right so that's why i put it all in a variable and then if the actual string changes later it's only one spot i gotta change it right so but what this is doing is it's getting it's going into that csv dictionary and it's going to print out um at least this it's going to get the value of the latitude field the longitude field and just print them out which i don't know if you guys noticed that when i when i first ran it in the beginning there was a whole bunch of stuff that kind of barfed out on the screen that's what that was what was happening there okay okay and then we have this um what does this mean try with its partner accept down a couple downline 70s anybody know right this is this is this is a basic what they call error handling right um if you're a good coder you should always be accounting for problems because whoever writes code that's perfect or heaven forbid you're relying on people humans to do things the way you intended them to do people never do things the way you want you know especially your writing so you have error handling and so what will notice to the indent right so i put on line 57 i have the try statement and that means it's going to try to do everything now that's been indented over from that line and if something happens if it bombs line 70 is going to catch it and it's going to print out the error that happened okay this is perfectly written code with nothing that's going to go wrong everything's going to work fine so we don't we don't need the exception well at least so far it's worked good um and so that's so that this is now where the real processing happens um so what this is anybody know what kind of variable i've set up here we just talked about it how about what's your name ethan what's that what is this variable here what did i set up here it's a list that's the answer a list yeah so it's a list of all the field names and this is a setup for getting ready to write data into the shapefile okay and if you were curious i did hard code this this is just an esri thing this is how you reference the name of the of the of the shape column in the geometry i don't know why they do it that way that's just how they do it um but here's our friends we had from before right the record id over deal overdue field name sr and in the geometry and if i go back to arcgis pro that's you know their shape record id overdue sr type i'm getting ready to reference those fields when i write the csv content in okay okay and then cursors any of you familiar have you ever done database programming and use cursors i don't know if that's come up one person anybody else come across cursors like a forward only cursor what's a cursor it's like an object that represents where you are and like where you can write yeah yeah well okay you never know when you guys might go on a co-op or come across this if you ever really have to work with data like you have to programmatically access a database cursors have been around even back when i made my living doing coding i mean long before i was a professor basically you can think of a cursor as an object in your code that can hit hit a table right so let's say i want to programmatically with code access this csv file my cursor object will allow me to you know delete records out if i wanted to do something like that it'll allow me to add records in and so forth and um they're like i said they're common in database programming and the same concept exists in arc arc pi because data fundamentally drives gis software we're just doing a shapefile but there's a much bigger universe out there of like enterprise geodatabases huge i mean imagine huge massive systems of spatial data you're the us government and you have to have every i don't know something like every school in the country in a database or every road in a database massive huge systems that run this kind of stuff um and so cursors are how you go about doing that in fact i think the gis miner still has a course in like just databases as one of the electives for just that reason because these things come up they're good concepts to know if you're going to really work with data anyways cursor on line 62 basically what this is doing is getting ready to write data into the into the feature class so it's an insert cursor object that's being created and we're telling it we want to go to this feature class that we created and still is in memory back from line 20 and notice by the way when i highlight it it gets highlighted uh i highlighted it down there it gets highlighted up here and then my fields which again i just put my mouse over it you can see how on line 59 the fields are getting highlighted so it's just kind of a vs code thing for referencing variables okay okay we're almost there now so now we've got to really just get get the content so what i do here on line 64. like the comment says i'm taking the latitude and longitude from the csv and putting it into a list of two float values okay and take a look look let's look at right about right about there can anybody tell me what's going on there with what i highlighted on line 64. there's a couple there's basically two things happening yeah absolutely what's another way i can call it convert was it yeah even the more that's the i am i'm converting what's another term casting right that's another thing of casting because if what do you think if left on its own if i just accessed that row long field what kind of data type do you think it would be it would be a string right it's just a csv file it's just it's text so you have to tell it even though it's a numerical format right you've got to tell it to turn into a float into an actual number as opposed to a string it's it's represented as a string in the csv and you have to cast it into a float okay and that's what we heard here you know we're going to create a list variable xy that converts the latin long fields into floats and kind of gets that ready okay and then line 67 is really now where really everything's going to happen um we're going to insert a new row using that cursor because we just learned that cursors are used for accessing data databases tables or whatever and it's going to insert a row and then now here's the fields we've seen before that row id value val that's right up here when it starts the loop it's at zero um i do another casting of the overdue field because that was um where was it right here that was um that's also a float a decimal point number again it's a string but i want to make it into a double and then um again you're seeing why now i um i didn't hard code my field names in right i'm just i guess i didn't really talk much about this this row you know that is a dictionary there's a dictionary value so i'm just using the the dictionary again is keys key and value so i'm using the key that comes from row overdue field to get the value in right where's that so it's basically like you know that like in this example this is the key and then that's the value right so i'm using that basic id basic idea because all of all of these column headings become keys and so i've got that key set in the variable and then as the loop is happening i'm saying okay what's you know i want overdue and then whatever the current value of the overdue is take that and put that into my into my uh my um my shapefile or really feature class okay and you can't quite let me um let's see yeah so i'm going to write the row id eval one record one column overdue second the type was just a string so i'd have to do anything at and then my coordinate value um which is the shape so everything on line 67 corresponds to the fields i put together on line 59 and those the order they have to be the same if you don't you i mean you it might work but if you try to put a string into a double or something it'll blow up you'll get an error okay all right and that's it that's the main writing of the records we increment the row id value by one using the plus equal operator and then um if no error is happening it goes back around again in the for loop and does the whole thing over again with the next record right so it's just gonna loop through the whole csv file run this code and then when it's done at the very end why do you think um why do you think on line 75 i have that all the way on the left what's that yeah it's outside the loop the loop is done right so i once i've looped through everything then i have to delete the cursor out of memory um because if you don't do that the shapefile will still think it's in edit mode and it'll act weird so that's kind of like kind of the equivalent of hitting like the save button when you edit something and then just a little feedback and so forth so that was was that helpful to go through really walk through it like that was that help explain it okay again you could just copy and paste it but i thought you know to try to really understand there's a lot of new concepts coming out here let's run it and um okay so it's all in there now if everything works according to plan i should see the 311 shapefile get updated and have content and there should be little points on the map around houston texas and there it goes worked great so there's our print statements there's those coordinates the done there's my records there's all that content from the csv that got moved over into the into the uh the shape file it got created and so forth did you guys did you run it did it work for you okay so that's it for the content i wanted to to deliver in class today let me just now go over your lab for this week which if you hold on to this code you have a huge head start for the lab because the lab is basically doing the same kind of thing except with a different data set and a few little tweaks so if you're comfortable with coding you should find this no no no problem um but if you're new to coding um this will help you get get you started and so basically with the lab if i go to um i thought i had to open it no i don't let me um let me open that up for you here uh lab two should be on it's on my courses under labs okay right sound familiar csv parsing with arc pi so basically what i want you to do we looked at 311 data but i want you now to read a different csv file in that's based on earthquakes okay and instead of you know here are some different kind of field names i want you to create so basically you know the big hint here is just use the code i gave you swap things around a little bit and should get it to run although you know i find that even with basically giving the code away students still run into problems so that's why i think this is a good uh a good learning experience so you know open earthquake csv parse it into earthquake shapefile um something i haven't showed you we'll go over that manually apply a graduated symbol filter i used to have code where you then change visually change the appearance but i i took that out i thought this would be enough for where we're at and so here's your deliverable again is your code and a screenshot similar to what you just did for lab one and so this little should tell you this is what it should kind of look like if you got it to work correctly right and then to do a graduated symbol filter if you haven't seen that i can do it with uh this right here so for example if i use the overdue what overdue means in this data i believe it means how long it took for the the three one one call came in then how long they were able to then the time it took to like respond to it kind of think like a you know like a ticket system like you call for tech support and they give you a ticket so i think that's what these numbers mean so i took them i think if i'm understanding that correctly you know they got to it way ahead of time versus right on time but it can actually make you know for visualizing it if i were to go to 311 double click on the symbol now it'll bring up the points but if i click the arrow here and then it asks you for a graduated [Music] graduated symbols that's what that means manually apply a graduated symbol render using six six class breaks so i just did that to make it six class breaks just make it six and um you know that's that's the idea there with that that part if you haven't just that's just a little software operation like i said i used to have where you would we would do that part of it programmatically but it was way more to try to fit it into one class would be it was too much so so that's the lesson for today i'm going to stop my screen recording um [Music] hi this is brian tomaszewski if you enjoyed this video be sure to like comment and share this video also please consider subscribing to this channel and clicking the notification icon to stay up to date on new videos from this channel thanks for watching

Channel: GIScience

Views: 41,109

Keywords: GIS, GeographicInformationSystems, Geography, GIScience, ArcGIS Pro, Esri, GIS Software, GeographicInformationScienceandTechnology, GIS&T, Maps, Mapping, Training, Education, Computers, Software, Python, ArcPy, VS Code, Computer Programming, Beginner

Id: 6Y9hAB5yYWo

Length: 76min 16sec (4576 seconds)

Published: Mon Sep 20 2021