How to Download Stage-IV Rainfall Data as NetCDF4 File and Make Time Series

hello there welcome back welcome to my youtube channel so today i'm going to show you how to download space for precipitation data and i'll also show you how to create time series from this precipitation but this time it is a little bit different than previously what we already uh seen right the previous net cdf file was only one file and it has like time chunk but this case we have only one file for each time and step then we have to do the processing in a different way so i'm going to show you how to download that data first because the benefit of using this state sport precipitation from this website that i'm going to show you right now is because this data is already in net serial format but the previous data set either in mrms or stress three or stage two these are all are basically in uh grip permit because that grip to our grip one gridded binary that is a pretty complicated but yeah sometimes it is useful as well to store a lot of information into a single file and within a very small memory so that is the benefit of using a grip file but this website it is you also have this stage for precipitation data in group format if you are in interested in that one you can do that but here i'm going to show you the process data it is already processed and converted to net cdf and file still the file size is pretty uh small and you can use that one as well so i'm gonna download this from this aorc nws if you just source this one you'll see the analysis of record for calibration so that is the website if you search and you'll be here and then if you go to this one index or aorc history then they all just direct you there and you have a lot of options here because these are from basically that rfc river forecasting center and there because i'm gonna download this data for the wz rfc okay west culp river forecasting system or river forecasting center here okay because i'm gonna download this one for texas this area i'll show you the aerial extent if i click on there and it is going to show even temperature if you want to even download the temperature data you can even do that because see we have this data from 1979 right from 2nd of february or from february but if you want to download only the precipitation these are tmpr temperature data if you want to download precipitation or rainfall that's that you have to go there right at the very bottom click there and we can see we also have the same data from 1979 right february and then we have after today here you can see this data but if you want to since i'm gonna even extract the time series i need to show you the plot so that you can understand that okay we are extracting the time series or we are generating the time series accurately so that if in order to do that i can even download this data for a shorter period of time for example i'm going to download this for harvey then you can see we'll have a very high peak and i don't need to download all this data for entire year and you can do that because we have this data the monthly file in a single month these data are basically hourly data and in a single g file you will see we have the led and i'll unzip that one and see that so i'm going to download for harvey 2017 and here from because harvey basically hit the coast of texas and louisiana uh like at the end of august so i'm gonna download this data for august and september okay these two files if i click there then file size is pretty convenient you can see that it's pretty convenient see files are the 16.8 megabyte and it's covering the entire thing and i already have downloaded this one there so if you click there you'll have this and if you click there for september and you're done it is also pretty convenient so once you have this data downloaded data there right here so what you have to do you have to unzip this one and if you already have that software installed just extract and before extracting because it will make a lot of files here i'm going to copy and i'm going to create a new folder means stays stays four okay stays for precipitation space for precipitation and i'm putting it there this okay and i'm just unzipping here and let's see what we have and then i'll check the aerial extent and i'll show that before extracting and processing i have this one and then so these are the demerit of using that windows but if you use linux machine it will be very convenient to use that one so i'm extracting manually because these files are not that big that's why otherwise i use linux machine i'm deleting this one i don't need this and what i have to do see inside this file i have a lot of hourly files for entire month right for one month have 744 so what i'm going to do i'm going to even select and then ctrl x and i'm putting it there all together okay then i'll delete that one so for september i'm going to do the same thing ctrl a and ctrl x and i'm putting there so i have this data in a single folder right and then i'm gonna delete that empty this one so we're done see you can check the name of the file right these are the aorc that data and then wgrfc and you can see the date this is very important part for our case to generate time series 2017 august right first of august and zero zero hour so these are the year month day and the hour and we have this data all the way to the september 30 right and 20 23rd hour so we have this data each data will have several variables and let's see what variables we have let me just plot it plot it and i'm going to show you the aerial extent of ease of this file and you'll see and because without understanding the variables and aerial extent we can't do that right here we have i'm just opening one file and here the apcp surface okay this is the variable name and you can read that short a pcp surface we have the time one time the for it is basically three dimensional data the first state is the time and then we have the latitude we have 450 values basically these are the columns and we have the rows 495 and see the unit you can see kg per meter square this is basically the millimeter if you have a kg within one meter square area basically it represents millimeter right so if i want to plot it and you can see the aerial extent here it is that is the area we downloaded here see there and if i even try to make it bigger so for this case it is this and if your area is different you can say if you know your river forecasting drone then you can even go for that one see basically this is covering the texas coast of texas okay and louisiana as well here a part of this okay so we have this aerial extent and if you have your point anywhere within this region right within this region and we can see we can find one location right we can see and we can even if we have multiple location we can even do that but i can tell you some station location here and i'm going to show you this one this is only for demonstration nothing else the levels the map and here i'm going to give you that 0.5 degree and then i'll show you that every grid line and it should be 10 see it is pretty hectic right ok so i have to find out this grid or the coordinate if you know the coordinate of your study area if your study area right is within this limit you can find some stations that may be observation station and if you want to even compare with your simulation or your observation if you want to validate your model or if you want to even create the input file for your hydrologic model you can even process this precipitation data okay so how can you do that so arbitrary what i'm gonna do here i'm gonna even okay let me just try to find a good point or two so that we can understand see this is basically 98 okay 98 and if i select this point so i'm gonna write first one longitude okay longitude first one is minus 98 since it is west and if it is 98 and corresponding latitude is 26 north this point okay 26 98 26 and i'm gonna select few points two of them and then if i select this one it is 98 still 98 and for this case it is 28 and if i have another point there we can even select 98 30 okay 98 and 30 and if i have this point it is 98 98 and 32 and this is how i can go up and all the way to this one so this will be point i'll show you how to create this point and if you have one point there because we can see right we have a lot of information there as well but harvey hit this area so we can even select this point as well okay so what will be the value of this one it is uh 96 okay 96 and 30 and if you want to even click this one 90 7 another point will be 97 and it should be 29 okay this is the decision okay we can't even select more how can you do that so let's try i'm going to open a jupiter notebook because i have similar chord from my previous tutorial you can watch that one and then if it is not a related problem related to memory because this time these values are not that big we we can do that so in order to even see the variables right these are the steps i follow usually you have to import os and system if you need to even reproject because you have to check the projection system if your model is in different projection system and if your precipitation is in different ones we have to write convert it to the same projection system so and the main topic here i'm gonna use that nate cd4 module right to read any new cdf data because this is the net cdf4 so i'm going to use net cdf4 module and i'll even process some data using numpy array that's why i'm importing numpy as np and definitely i'll delete that time and data frame as well so that's why i'm gonna import this pandas as pd and definitely i'll deal with that time and time della i'm importing this one and i'm just suppressing the warning if i have any so these are the thing first tape i'm just creating a new okay creating a new copying and making it fresh here because i have several thing there and i'm just starting from there as header and this should be [Music] time series okay time series from stays for rainfall okay rainfall data this is the header and then i'm gonna paste it there i have this header there i'm just generating writing the code and when i write the code for the first time i is jupyter notebook and then i just use my linux machine or you can use even spider whatever you want so now i have this one so and then i need to even show one data then i look through all of this right so okay because i have this mouse that is pretty fast okay okay so we are here right we are here i'm gonna show you in a bigger font so let me just show the location where i have this data it should be there and i'm going to select only one file okay only one file to see the variables if you want to copy the path go to properties and security and copy it from there and you will have that so i'm gonna even put it there here you can see i have this and i'm just putting r as a reading mode right i have to use forward slash instead of backward here you can see that path is pretty long okay not a problem we have this data and we'll look through all of this and if it is okay so now what i have to do i have to see right have to see the variables right how can i do that because it is an object data object and then variables variables dot keys and if you do it this way it will show the variables we have a pc surface and latitude longitude and now we can even see what are our dimension of this data right variables and in the variables if you put that fecp you can see the information of this one okay and what is the unit and everything you can see the properties here you can see the missing values and you have the current shaft the first has to be time right it is there first one is the time and the latitude and longitude if you pass the index of time and the index of the latitude for your specific point and longitude then definitely it will give you the value and if you want to even import okay if you want to even import or save or show all these values of precipitation i'm going to show only four few of them and if you want to show this way it will give you everything okay let me just run it see it is going to give you this the first rows and columns are zero values so that's why it is there okay i'll store this one not a problem i'm just checking the variables i have this way and if i want to check the unit and variable for other like latitude longitude and time for example latitude okay let's see what we have it's similar the name should be latitude and is basically degree north it is already in degree right so we don't have to be a bit about about it okay so we are done with this one and time for time let's see if i want to even check time what is the unit of the difference time that is very important and here we can see the time seconds since 1930 and this is the time first of january of 1970 okay 1970 and the time we have it should be in second and if i want to print the values of time it will be amazed that it is pretty big number okay and i'm not going to show all this time i'm going to show only the first 10 values of time because it has only one shape so these are the values of time see can you see that it's pretty big number this these values basically it is in second and this second if you calculate that manually from the 1st of january of 1970 you will find this number of second so we'll use this second to calculate our starting date of our data because our data is starting from we know that right we can do it in two different ways because we know this data right you can see that the date is there we can even calculate our the time object using this file name because it is already written there we don't need to even start the same thing we did for era5 if you didn't watch that video you can go back to my previous video the last video i uploaded you can check that how right how did i generate that dead time object from this reference date but for this case since we already have this information there right in the file's name so we can select this part and then we can generate our date time so that would be easier way right whatever the sequence of the file whenever it will read this file whenever it will loop through this file it will get the date time and it will create that time now object and then we'll make time series out of this one so okay let's go for that so i think we are already done right we're already done with exploring our variables we are pretty good so now i have to specify so i have similar code there so what i have to do i have to okay so after doing that let me create the input or output directory these are the directory where i have to do that and here is okay so i'm specifying my output directory of my time series it should be within that okay it is going to be within our stays for precipitation okay it will generate one new folder by the name of uh precipitation or rainfall okay rainfall ts rainfall time series and it will check if this directory exists currently there is no name by this one rainfall ts so it will create one okay and then i'm gonna read the file name okay the station name basically because the location i just mentioned you here from this map because i don't know the location i'm just showing you if you have any location i just calculated the location right i'm going to give the name as station 1 2 three i'm gonna prepare a similar file because it it should be pretty easy i'm gonna copy the file and change the location name it should be within my era 5 there i'm going to copy this file and i'm going to put it there stays 4 it should be within my download right there here right see i can put it there or i can put it anywhere i can put it anywhere it doesn't matter okay even i can use the same value and i can put it there as well okay i'm putting it there and i'm gonna rename this file as station okay station means rainfall okay station rainfall and what will be the location of our station because i have many stations here i don't wanna waste time on that so for example i'm gonna give the name as this one and first one will be the serial and i don't need this all of this okay because i have to then i have to provide the location of all of this one okay i'm gonna just delete all of this and i'm gonna create the time series for 10 locations what i have longitude right previously i mentioned that one it should be minus 80 and the corresponding value was 26 right the name should be anything if you know the station name you can put it there and then the next one if you want to even generate the time series for the same longitude value 80 and then that should be it will increase right 26.1 for example this one i'm just increasing number or 26.25 and this should be minus minus 80 and then that should be 26.5 and here if it is a minus 80m this arbitrary giving so that it is within our study area or the data we have right so that should be 26.75 and minus 80 that should be 27 okay and then i'm gonna change the location from this one to that one it should be minus 96 and that should be 30 and if it is minus 96 that should be 30 okay 30.5 i'm just giving the name and then that should be minus again 96 and that should be 31 and then i'm gonna put minus 97 and that should be 29 and minus 97 that should be 29.5 and that should be minus 97 and this should be 29.75 okay this is the location i'm just saving this one i'm gonna loop through all these stations and generate the time series based on the name of the station right name of the session.csp okay so i just saved it and i'm gonna copy the location as well here right this is the station i have rainfall properties and then security and copy the name and the path as well and i'm going to put it there so that is i'm reading right in order to read any csv file you have to use panda right pd dot read unless coordinated event i'm gonna just create the index for the first column because i have that number and i have to put it that way forward slash and let's see if i can do it quickly maybe i'm almost done okay and then if you want to see the stations right because i'm going to print the stations stations if i hit run see i have the list of this station here what i'm gonna do i'm gonna loop through each of these stations right and then i'll create the file name the output file name and see by this time it already generated the rainfall ts see inside that file it's there see you can see that it's there okay i don't need that this one i'm just downloading this okay and i'm just deleting that one okay and then what next i have to look through so in order to look through we key and then value and that stations dot address okay a value in station dot writer rows and that is the function if you want to see what it's going to give you if you print key it will give you that 0 1 2 3 number if you want to print that stations then you have to use a value and this index should be zero if you want to print the longitude then the value and the index should be one and if you want to print that latitude it should be it should be index two okay so that's that so we have access to our this one and now if you wanna even create the file output file name okay what will be the output file name so i can generate this way formated string form format and then the value okay because this value and that should be zero is the station id then i'll have that dot csv i'm gonna export it as csv okay and if i see i'm gonna print the file name and let's see because i don't want to okay i can even print that way as well run it see the first value first one this is the row i'm printing both and it's going to create the name of the output file.csv right so for this station it is going to create these file names this is how we'll export and generate our time series and we'll save that in that rainfall ts okay so after looking through each of this thing and for our case we have this number of files right and then i i have to look through this directory and insert this file so how can i do that now i have to specify my input directory there as well now it's the time to specify my directory and i can do it here that is the output before specifying output directory i usually do input directory as well okay so that will be my input directory because here i have my files and there will be one problem because i'm saving that files inside this data file right so that's why what i have to do you see if i run this one if i run this one i'll have that folder there right this folder and next time i'll run it will even consider this one as a list of files in order to exclude that difficulty what i have to do i have to provide another condition here i'll only look through those files that has that dot nc4 extension right so how can i do that it's pretty simple it is not a problem so when i'm done with my reading that csv file and i have the file name everything ready for each of this file name i have to look through each of this net cdf file to generate time series right so how can i do that so here now i have to change the directory to the input directory or i can even see this is how i do things there i have similar code there for looping through each of this directory or folder um just gonna use that os dot list here this is how i do okay where is dot list here and i think i have this thing there as well wait a minute if it is not there i can generate extracting okay but i mean it maybe i have similar thing it is not there okay we are here right we are okay we are here now since i already imported everything i can write it no problem so for i'm looking to reserve this files from that directory input directory right for file name for file name in west west dot list d okay where is dot list here that is the function but what will be the list here that is the location okay it will list every files there let me just show you and then if i have to use if condition because there will be other files as well i'm going to loop through only that dot nc4 files so if file name okay file name dot ends with that is the command ends with what ends with this dot nc4 then it will only then it will do this thing and that thing okay i'm gonna print that file name and i'm going to show you that if it ends with nc4 then it will show this one okay i'm not okay i'm going to show you only the file name for his file name means for each station for this station it will show you everything okay that that's what it should show right now i'm running it see for each file means the station id it will save a csv file when i'll do that it is printing everything every file and for the next file name it will even repeat the process okay it is gonna repeat everything see within fraction of second we can't differentiate maybe it could be there anywhere there it is august 30 okay september september let me show you that september yeah when it's done with see when it's done with that september 30 and 23rd hour it is gonna move through another file we'll do that so we are okay so far okay now i have to read ease of this net cdf file and i have to extract the variable and i have to do some other processing then i'll be able to generate that one so for ease of this station what i have to do i have to create here the time okay dead time empty list of the time i'm specifying that because i'm gonna create that one and then i'll even the data or the rainfall okay rainfall empty list i'll fill up this empty list and then i'll generate data frame and i'll save it as csv file that is our goal for today right so we have this file and then i have to read in order to read any new serial file so that is the criteria because if you want to even run that right if you we use net cdf as data set that's what i did previously so i don't need to even do that near cdf.net cdf or you can even do that import near cdf and you can even follow this this one as well okay it doesn't matter i'm gonna copy since our file name is there if it is ending with this name so what will be the value of data object for this case it will be file name and r okay so this file name if you just copy because it will look to rest of this file and it's gonna read and i have the date time or data object and i have access to every variable now what i'm going to do here i can put these variables data dot variables and precipitation this one okay i'm going to do it here when i will read this file then temp or tempressive or passive okay whatever it is press it is equal to this apcp and i'm gonna export all these values now i have every values the precipitation values right i have that now what i have to do if there is any negative value there maybe then i have to take care of that values as well so i have similar uh thing there i can even show you from my processing for the same case i did i just i can because i tested that one and i'm gonna just show you this uh pretty quickly okay that doesn't matter so just let me quickly show you okay i'm just uh keeping it aside and whenever it is necessary i'll do it okay so once you are done with that one so you have to just make some treatment there if it is necessary otherwise it is okay if there is a negative value because we don't want to use our negative value right so we have this variable passive and then we are going to replace it if the pressure value is less than zero means there won't be any negative precipitation negative rainfall right there so we are replacing these values with zero then what i'm gonna do i'm gonna read our latitude and longitude all this latitude and longitude like the way we just import or extract this apc data so longitude will be this longitude and all this longitude will be there as long and ladies will be there as lot and then what i have to do as before if you watch my previous videos i just do this thing i'm going to make another array of square difference in between my station the station i'm gonna give and the station right on all this longitude here i exported longitude and latitude right but before that we have to export our lat point or long point right for our case we have to be able to export that one so how can i do that i have this values latitude and longitude for my station and the left point here the left where is my left point i have to use the lat point here because for my case i have the similar chord just if you go up here if i is i can use even similar logic from this one as well the previous video i have it is here because i have the station name here right and then what i have to export i have to export the latitude along to this latitude this latitude and longitude so if i use one index it will be long point and lag point because we are going to export this data for this longitude and that lets you in order to do that what i'm gonna do i'm gonna just show you one more time the same thing i did because the data i have it is like this in grid right like this grid i have we have this is the longitude long okay this is the longitude along eastern west and i have the latitude as column this way basically what i'm going to do because for example i have this minus 80 and 26 there somewhere there close to here okay so i'm going to make the difference because in the data set i have this latitude longitude i have this relationship longitude this and for all this point i'm gonna make this difference between this point and difference between this point this point and this point at every point every point everywhere i'm gonna make this difference okay and this difference may be negative or positive i'm gonna make all this difference for every point right every point i have for all this point from my data set and from my in between my point as well location and i'm gonna make it a square so that it will be positive right if i square all of this the value will be positive and if i can consider the closest value from the data set as my point value because otherwise there is no already way because the data right it is either in this node or this node so i have to find out the closest i have to extract the closest value from the data set so that's why i'll find the index of the minimum this is the minimum distance so it will represent this value this point will represent this point so i'm going to extract the data transfers data for this point and it will represent my point so that is the thing i just did here right i made the difference and made it square and then i'm gonna find the index index means what is the number of this latitude because if it starting from right if it is starting from this one so zero one two three the index of latitude will be three and what will be the index of the longitude if this is zero zero and one so this point will be uh like uh three right latitude is three and longitude is one if i specify let us see this three longitude is one and for this first time step it has to be one means the index should be zero if i pass the zero three one the time index is zero the latitude index is three as long as your index is one then it will represent this point and it will give you the value of the precipitation okay so that is the logic i'm gonna use here okay so i think yeah you understood that one so that is the difference i'm gonna make so long point in between all this latitude of that data and the left point and all this latitude from that data and i'm giving the name square difference latitude square different longitude that is the point it is not all left okay and i'm gonna even export the longitude and latitude index of this latitude and longitude from this one so what is the command for that one it is arg mean and arc max if you want to even get the index of the minimum value from any array that is the command argument right so i'm gonna get the index of the minimum value of the square difference i'm putting it there minimum index latitude minimum index longitude it will give you that one and we know the time is right this file has one time instead and if i want to even print okay the precipitation for this case okay let me just show you the print for my first point uh print that press if so what i have to pass past zero index for the time and then the latitude that is here it will print the latitude right and comma that longitude it will give you the precipitation right it will give you the precipitation but here okay so no sauce file in the directory because we okay we are doing different thing there okay not a problem because we have another file here a pcp we have the file name we're almost done right impressive data longitude latitude trial line number 12 here here line number 12 data net cdf oh here okay net cdf dot that is the reason i have to put everything all together then it will work i'm gonna copy this one and i'm gonna put it there this is the time to make or compile all this thing together and i'm gonna copy then it will run all together okay so at the very beginning here it is okay so here i'm just modifying the kernel i'm clearing everything and let's start from the beginning okay everything is done so i'm importing this one here is my input directory as my output directory and if this directory exists it will it on create otherwise it will create and then i'm reading the stations and printing the stations i'm listing that time rainfall and looping through each of these stations and creating the csv file name and that is my long point is my lat point okay and then i'm looking through the directory and for this case what is the best way to do that is the best ways hd changing directory to input directory that is the best way to do c os dot change directory okay every time it will look through this one it will change the directory to this one then i don't need to put it this way and then it should print okay it should do the job for us if everything is okay so it's printing the value of precipitation and the file name as well see it's printing it's pretty big it will take a lot of time to print we still have the value of precipitation zero zero zero zero zero for this case okay maybe we'll have some values when the harvey that will come but i don't need that one the code is working until then okay so that's the point so we are done with this and now what we have to do we have to add this value to my empty list there the rainfall right this one so i can put it this way as a variable or i can directly add this okay add this rainfall rainfall dot append and i can put it this value so it will create okay it will add this value of precipitation for that specific time period to this rainfall empty list then i have to even fill up the empty date time list okay how can i do that when it will read is file because we know right the name of the file we know it so we have to split it let me just copy one file and follow the rule i did previously so here it is is filename is this one so filename okay when it bring the file name the filename is dot nc4 and that is the file name we have i'm just making it as command comment so what do we have to do we have to split the file name right how can i do that when i'm gonna split the file name that is the command i think i already have it here or i can even repeat that one not a problem so i think i have this type of thing there similar thing i did maybe let's check here it is okay it is different one but here okay i'm gonna just split it i'm gonna split the file name using okay first one i have to do this way this is the general code because i have this card from my previous tutorial i'm just doing the same thing see here i'm just splitting the file name splitting it because the format you know the first is the aorc i'm splitting it using os.path.split text it will split in between split into two this will be the first part and dot nc4 that will be the second part as the file extension and since i'll have this part okay this part and i'm just splitting it using that underscore so the first one is the aorc a pcp and wgrfc and this is the important part i need right this is the important part so i'm giving is year month day and hour right so this will give you year month day and hour if i just print it it will give you this string value so from this string i have to create that time object so here it is right i created that one that time object it is not a no problem i think i already have that here so what do we have to do you have to specify okay you have to wait a minute where okay maybe it is there okay wait a minute so that is the format if you pass that dead time and strap time you have to pass this way it will give you that okay there is the command we can use it will give you the dead time i'm gonna show you right now here it is okay i'm gonna print it print that dead time because i imported that dead time and strap time and it's gonna print okay because the string i'm gonna pass the first one is the ear that's why it is the notation and then may month and then day and then hour it's going to print this date time our everything okay so the file name i don't need to print this file name i don't need to print the value of the precipitation this time i don't need to do that it will only print the date time okay this date time and file name as well okay let me just check it see the first one strap time that time has no attribute strap time because we have to import the time the time we don't need this one okay because from that time i'm importing the time dot time delta see it's working so here it is for this case we have this file right now reading the file this file this file okay now reading when it will read the file this command it should be this file name so that you can understand that it is reading is including that file so for for this file it is excluding that uh extension so it is creating this data entry 2017 1st of august right august 1st and hour 10 right that is the hour 10. so we are done with our date time so what we have to do we have to add this that time after creating that one this object to our empty date time okay that time so how can i do that that is the thing here it was just this dot append and if i add this it will populate this one okay so we'll be okay we may have to even kill our kernel because it is still running we don't need to do that okay so we are done with adding we don't need to print every time okay or we can even print it is not a problem we don't need to do that it will populate our empty date time there and at the same time i'm populating the rainfall okay from here now it's time to save our csv before doing that we have to generate a data frame how can i do that we have to generate our data frame and here is the code i already have similar code right we'll have this data frame from this one when i'll done okay i'll be done with his file name so i have to go back to i have to go back from that for loop and here i have to paste it and that is the end of the loop for each station then it will start looking through each of this thing and then that range and that time okay i have this i don't need anything i don't need this one as well it's a this will be this i don't need because if we have your starting and ending time we usually use that for this case i'm creating this df pd.dataframe and i have the list of datetime index or datetime object and the column name is datetime and i'm just making this as my index okay that time index then it's a time to add my rainfall data and that will be the name of the header right rainfall rainfall rainfall this way rainfall and what will be the unit we know it's millimeter right rainfall that is the millimeter and since data is in basically hourly so we can put it this way millimeter per hour that is the hourly precipitation and i have to add the value of this rainfall when it will be done then it will add and then it will print okay it will print that df and see the data from there df but we don't need to print anything not okay now reading this one this one we don't don't need that okay we can print the file what it is reading all looking through this one station name okay we can try and then we'll save it okay let's run it see it's looking through this one and when it will be done for the first time definitely it will show you the data frame if everything is okay it's gonna show you the thing hopefully it's gonna show you that it's reading see it is reading after august 18 right maybe it's out of memory i don't know it shouldn't be out of memory okay let's see if it's continue running definitely it should there is nothing i can see there is no error bye-bye we don't need to restart our kernel that kernel what is showing that kernel or we can run it in my linux machine as well if there is anything wrong i'm hoping maybe okay wait a minute this is the problem within your windows machine sometimes it will show you this thing that thing okay not a problem so i'm gonna finish off this when i'm done with creating the data frame and i'm gonna save it right now it's the time okay it's the time to save and here it is okay when i'll be done with that one then i have to save it in my output directory the following time series the time series i'm talking about that time series i don't need that one so it will be saved there what will be the file name this will be the file name right file name and it will save everything there did out and then file name means just output file name until 7 inside that rainfall if everything is okay maybe this kernel is dead let me just interrupt again restart and clear if it works okay now everything looks okay otherwise i'll run it in my linux machine because it may be because of our memory issue sometimes this happens it's reading reading reading even i don't need to okay i don't need to show that okay it's reading this file of that file appears to have died maybe okay man okay so here it is it is maybe out of memory or something like that i don't need to show anything then i don't need to show anything i'm just repeating the process repeating the process i'm done let's try now if it's gonna work everything looks okay so far okay yeah let's see it will only show you the print when it will be done with the data frame okay that trend is not defined or that's what i was expecting here here we have to change this name to date time that's what okay um again clearing everything these are the steps you need to know because i'm testing this script for the first time that's why because i know how to do that and let's see if it's gonna work otherwise i'll copy the entire script and i'll put it there in my linux machine see it's created the data frame can you see that yeah for this data frame and it's working for another one the time and the date and i'm going to show you this inside that directory it should be there see oh the file name we need it is different one okay wait wait the file name i put it it should be the csv file i created the same name so that's why it was station name okay so here i should rename it because this file name and this file name is the same that's why i have to create this as name station name then it will be done station and this should be station name and i'm gonna kill it again okay i'm gonna kill it and let's try definitely this time it's gonna give you everything i'm gonna delete this entire directory and okay hopefully this time definitely it's going to create then i'll show you the plot as well after this one i'll show the plot because you need to show the plot as well right so it's going to give you this one shortly see we have this time series and definitely it's going to give you csg file this time see this is the csv file you have and this is the dead and rainfall and millimeter right and i don't know the values maybe for the first time instead of for this station 0 0 and i'm going to show you the plot if there is any so there is nothing it's zero for this station zero millimeter okay well decision maybe for other station i'll have some data or not let's try see i have other stations as well still i have this zero value the value is still the precipitation i have for this station there is nothing because it is outside of our precipitation region because i selected arbitrarily i can even change the location not here okay maybe i have to change the location let me try for another station because this is far away from the harvey area that's why there is nothing significant rainfall it was in different place not there because i get it i get some values there you can see here i can see a small amount here let's say because i'm just arbitrary selecting this location but i can change the location uh from these two this one okay find the another one because it is texas it should be somewhere 98 there it should be here maybe this okay if i select 28 and 100 or even this location okay this location would be even perfect if i select this location that location ultimately this is the way we can even select since we can see we have the values of the precipitation there right so we can select 34 we can select 34 and okay this area 34 or this area 30 30 and you can select this 100 30 100 okay 30 and 100 so instead of 80 say 80 it is far away from that one 100 and this should be okay this should be okay 26 i'm just changing the values on red to show you only 100 and this should be 100 and this should be like minus 95 and 27 arbitrarily okay 97 30 okay let's try with this one if we have something i saved it again i can run the code and let's try but before doing that i'm gonna plot at the same time so that you can understand what i have here right if i plot it at the same time here using this code because i have similar code there then we can see here instead of changing all this together okay i'm gonna create a subplot and this will be this thing and i'm gonna plot the index and this will be rainfall rainfall and what will be the title that will be rainfall okay rainfall and it will show you everything there okay let's try if it's gonna print something or not definitely it should print something for this case since it is running a station is not defined because i'm using that station instead of station name it should be station name instead of station okay it should be station name this station name this station name because sometimes i use different code so here i'm gonna run it again and see if it's gonna work definitely it should work if everything is okay otherwise maybe i'll have other issues here i don't think so that i have other issues but plt okay i have to even import that plt because i didn't import that plt so i copied that one and i have to even do this thing okay plt in order to plot anything you need to use matplotlib okay and it is matplotlib this time this is the modification you need and since i'm writing this code and testing in front of you so this is how i just write code and test in jupyter notebook and unit is not defined because we don't need that unit as well i use this unit i i don't need that one it will be millimeter and i don't need that one okay okay wait instead of doing that i can use that millimeter here okay instead of that i'll be using this millimeter then it will put it there and i'll have this thing everything notice these are the small details you need to change because if you copy this code similar code from another location then you have to change the variables and maybe this time it's going to work see it's plotting see can you see that yeah it's plotted so now we have the precipitation value see the value is like 17.5 millimeter there right 10th of september and you can see september 8 here right this location is a little bit away from and it's working for other station you can see that right this is the precipitation rainfall and then it's going to increase these are the values we have 25 for this case it is increasing and yeah these are the time series and it's saving at the same time so this is how we can download this aorc right this data that is very processed already processed you don't don't need to even convert it from grip to neat cdf that is the benefit we have and it's gonna finish off see it's increasing see when we have this location i guess 22 to august 30 the value of the precipitation is 70 millimeter per hour that means this is the harvey you can see that you can check it harvey this is the harvey see harvey is 70 millimeter per hour and it is 70 millimeter per hour so you can see huge amount of precipitation right what's dumping there or that area coast of louisiana and texas and you can see all of this in more than 70 events see this is where we have this event storm event harvey and massive inundation was there it flooded everywhere right and it costs a lot of money so this is it i think if you have any query you can put it there i can try by yourself it's pretty simple i'll give you the link of the download data in the description below and you can download it for your time period because it is extending from 1979 to the recent net and that's the very important and useful data source especially for precipitation if you are a new user of rainfall so you can process right your rainfall from net cdf file and then you can even convert your degree file if you want but that's the very important part dealing with precipitation you need to learn the data format and that's it okay i'm not gonna even elongate the length of the video i'm gonna finish off here and you can suggest me or if you have any query or you can even if you need anything you can even put it there i'll try to make those tutorial as well because other people they're asking for something i'm trying to even generate some kind of code based on their requirement my intention is to help other student and learn new thing this is the new thing because sometimes i work on my research for different cases but if someone asks to do this i learn new thing i because i like to learn new thing and i like to take challenges so that's it thank you very much for watching these videos and thank you very much for supporting bye stay safe

Channel: Md Arifur Rahmahn

Views: 233

Keywords: stage-iv rainfall, how to download rainfall data, time seiries, latest rainfall, harvey rainfall download, how to read netcdf4 file

Id: 5luR20Di9EY

Length: 63min 42sec (3822 seconds)

Published: Wed Sep 29 2021