Forecasting Weather with Neural Prophet and Python

i wonder what the weather's gonna be like today hot i hope maybe you should check the forecast what's happening guys it's super sunny outside but will it always be in this video we're gonna take a look at how we can forecast the weather using a time series package called neural profit let's take a deeper look as to what we're going to be going through right so in this walkthrough we're going to be going through a couple of key things but specifically by the end of this video you should be able to predict the temperature using the neural profit package so first up we're going to start out by pre-processing our data and loading it up so we'll be using a kaggle data set for this we'll then train a time series forecasting model using neural profit so this is actually a library that's been built on top of ar net and facebook's profit package then what we're going to do is we're going to forecast the temperature into the future using our trained model now let's take a deeper look as to how this is all going to fit together so first up we're going to start out by reading in our data set into a jupyter notebook using pandas we're then going to do a little bit of pre-processing specifically you'll see that in the data that we're going to be working with there's a missing segment of data so we'll take a look at how we can handle that then what we'll do is we'll fit our neural network model specifically using neural profit and then we'll be able to forecast into future periods so this will allow us to forecast the temperature out into the future ready to do it let's get to it alrighty so in order to go on ahead and predict the weather we're gonna need to do five key things so specifically we first up need to install and import our dependencies so this is largely the neural profit package then we need to read in our data and process our dates and i'll show you where to get the data from we'll then train our model forecast and then save our model down so we can use it again later on now the data set that we're specifically going to be using is this rain in australia data set but we're specifically going to be predicting the temperature so there's a column in here called temperature at 3 pm and that's going to be the feature that we aim to predict now another thing to note is that you can get all of this code pre-written so including the jupiter notebook and the data set from my github page so if you've got a knickknack knack forecasting weather with neural profit you'll be able to pick this up and run with it but in this video we're going to go through it step by step so first up let's go ahead and install our core dependency which is going to be neural profit alrighty and that's neural profit installed so in order to do that we've written exclamation mark pip install neural profit and you can see it's gone on ahead and installed it now the next thing that we need to do is actually go ahead and import some key dependencies so the specific dependencies that we're going to need are pandas neural profit matplotlib for a little bit of plotting and we're also going to import pickle because that's what we'll use to save our model down to disk later on so let's go ahead and import those all righty so we've written four lines of code there and we've imported all of our dependencies so the first import that we've done is for pandas so to do that we've written import pandas as pd then we've imported neural profit so to do that written from neural profit import neural profit so this is importing the neural profit class then we've imported matplotlib so from matplotlib import pi plot as plt so this means whenever we refer to plt we'll actually be referring to pi plot and matplotlib and then we've imported pickles so to do that we've written import pickle now the next thing that we need to do is actually go ahead and import our data so the data set that we're going to be using is this weatheroz.csv so if we actually take a look at the preview you can see that there's going to be a whole bunch of different columns that are really revolving around the weather that's taking a little bit longer let's just read it into our data frame instead so let's go ahead and read it in using pandas alrighty and you can see that we've now got our temperature data so in this particular case what we've done is we've written pd dot read csv and then to that we've passed through the name of our data set so this is our data set here so whether dot or whether oz.csv and then we've stored that data inside of a data frame called df and then in order to visualize the first five rows we've just written df.head some pretty standard pandas functionality there so if you're not too familiar with pandas i highly recommend you check out the pandas in 20 minutes video i'll include a link somewhere above check it out it's really good now that we've imported our data it's probably good to do a little bit of exploratory data analysis so you can see that we've got a whole heap of different locations here and it looks like we've got a number of columns so let's take a look at those first so let's take a look at all the different locations that we've got all right so you can see that we've got quite a number of different locations in our data set so to get this unique array what we've done is run df.location so that's referring to this column here so if i drop the unique you can see that we're just pulling through that column but by typing in unique we're able to get all the unique values within that column now you can see that we've got quite a fair few the one that i think that we'll probably do is melbourne so you can see that we've got a whole bunch of different locations these are obviously australian locations we're just going to pick one so feel free to choose a different one if you want we'll pre-process this in a second to filter this out so we're going to probably do melbourne now the next thing to do is probably take a look at all the columns that we've got so let's go ahead and do that all right so we can take a look at our columns just by writing df.columns and you can see that we've got a date column location minimum temperature maximum temperature rainfall evaporation sunshine wind gusts wind gust speed direction a whole bunch of directions as well as wind speed humidity pressure whether or not we've got cloud at 3 pm and 9 pm a temperature whether or not there's going to be rain today or rain tomorrow now in this particular case we're going to be focused on the temperature so we want to forecast the temperature going forward and i think what we'll do is we'll forecast the temperature at 3 pm so towards the afternoon so these are just things to note so we're going to be doing melbourne over here and we're going to be forecasting our temperature at 3 pm now the next thing that we need to do is actually start doing a little bit of pre-processing so we want to filter out and get a specific location and then we also want to convert this date column to an actual date time so if we actually look at our d types at the moment so by typing in df.d types you can see that our date is just an object so it has no time properties but we want to convert this to a date time object so we'll do that in a sec as well so let's go on ahead and start pre-processing our data set so what we're going to do is convert our date column into a date time type and then what we're also going to do is we're going to filter out our data set and just pick out melbourne so remember we're only going to forecast one location so let's go ahead and do it alrighty so we've now gone and filtered in our data set and we've converted it to a and converted our date column to a date time so if we just check our d types uh that doesn't look like oh hold on we're doing it on the wrong data frame yep cool all right so that's done so what we've written there is we've written three lines of code so the first one is we've gone and filtered out our data set and specifically grabbed at our melbourne location so to do that we've grabbed our data frames our original data frame which was df and then we've gone and applied a filter to it so specifically what we're doing is we're grabbing that location column and we're only bringing back rows that have a location of melbourne and then we're storing this filtered data frame inside of a new data frame called melb so going forward our core data set that we're going to be working with is going to be called melb so m-e-l-b then what we're doing is we're converting our date column so specifically what we're doing there is we're grabbing our date so melb and then we're passing through our date indexer so this gives us our date column and then we're using the pd.truedatetime method and we're passing through that date column so this will take our date column which is just an object and convert it into a date time type then we're storing that or overriding the existing date column with that new date time type column and then we're showing our first five rows using melb.head so you can see that we've now got our first five rows there and if we type in melb.d types you can see that in fact we've now got our date time or our date column converted to a date time type so this is a requirement whenever you're actually working with neural profit you need to pass through two columns so the date time or a date column and then a value column but we'll see this in a second so now the next thing that we probably want to do is a little bit of exploratory data analysis so we haven't actually looked to see whether or not there's an actual pattern in terms of our temperature so what we're going to do now is we're going to use matplotlib which we imported up here and we're actually going to plot out our temperature over time so let's go ahead and do that alrighty so you can see that pretty clearly there's a bit of missing data here so it looks like from 2015 to maybe mid 2016 it looks like we're missing some temperature data there so we ideally want to make sure that we don't have missing values whenever we're passing our data to neural profits so you can impute certain missing values but if you've got a large portion missing then it makes things a little bit difficult so rather what we're going to do is we're just going to cut off our data set from 2015 and just use 2015 onwards or prior so that will basically mean that everything after 2015 we cut off so everything from here onwards is removed from our data set so let's go ahead and do that alrighty and there you go so we've now gone and filtered out our data set so you can see that we now only have 2015 and prior so in order to do that we've done it in two steps so first up what we've done is we've created a new year column so if i just go and type melb.head you can see that we've now gone and extracted our year so if we go and look at the tail bit we should be able to see the bottom so what we've first done is we've gone and grabbed our year from our date so to do that we've written melb and then pass through our data index our so this gives us our date column and then we've gone and used the apply lambda sort of functionality to be able to go and extract our yelp so written dot apply lambda x so this loops through each value in the column and then we extract our year so to do that written x dot yeah so this column here is basically looping through every single value in our date column and just grabbing the year and then we're storing the results inside of a new column called year and you can see that we've now stored it in a column called year over here then what we've gone and done is we've gone and applied a filter so again similar to what we did to filter out by melbourne location so here we filtered by location and we grabbed melbourne here what we're doing is we're filtering by a year so what we're basically saying is that we don't want anything prior to the end of 2015. so this will capture everything prior to 2015 2015 inclusive and then we're filtering that out and we're basically grabbing every single column inside of our melb data frame and we're overriding our existing melb data frame with this filtered one so you can see now that we don't actually have anything past 2015. then what we're doing is we're reusing this visualization code that we had here so we're using our matplotlib functionality to go and visualize our plot and you can see that there that we in fact don't have anything after the end of 2015. now the next thing that we need to do is filter out a couple of our columns so specifically whenever we're passing our data to neural profit it's expecting two columns and two columns only it's expecting a column called ds which stands for dates and it's expecting another column called y which represents the value that we're trying to predict so ideally what we're going to do is set our date column to equal ds and set our temperature 3pm equal to y and then we're going to get rid of all the other columns so let's go ahead and do that now alrighty so that's our data filtered out so you can see that now we've only got two columns so ds which is our dates and y which represents our temperature at 3 pm also representing the values that we want to predict so in order to do that we've gone and filtered through our data frame so specifically we've grabbed our melbourne data frame or our melb data frame and then we've passed through an array which basically means that we're only going to return the columns in that array so in this case we'll pass through date and temperature at 3 pm and we've stored that inside of a new variable called data so this means our new data frame is just called data so if we actually take a look by typing in data you can see that this is now going to be the data frame that we pass to neural profit then what we've done is we've dropped any remaining n a value so or any missing value so to do that written data dot drop n a and then we've passed the in place equals true keyword parameter this means it will apply it to our existing data frame then we've renamed our column so to do that within data dot columns and then we've passed through an array so in this case we've named our two columns d s and y so the previously called date and temp 3pm we've just renamed them to ds and y because this is what neural profit is going to expect and then we've shown our last five or our top five rows using the data dot head method all right so we've done quite a fair bit of pre-processing there now it's time to get going with neural profit so first up what we're going to do is create a new instance of our model and then use the fit method to go on ahead and train so let's go ahead and do that all right so before i run that let's quickly run through what we've actually written there so we've written two lines of code so the first line is creating a new untrained neural profit model so what we've done is we've used our neural profit class which we imported right up here so remember we imported from neural profit import neural profit so we've imported that we've created a new instance of it so by passing through parentheses at the end and then we've saw that in a variable called m so our model is effectively going to be stored in a variable called m then what we've done is we've written m.fit to actually go on ahead and train our model now to do that we've passed through three parameters so we've passed through the data frame that we want to work on which is this the one that we just created up here with them set the frequency of our data frame as well so in this case our data frame is in a daily frequency so you can see that it's got each day and then we've specified how long we want to train for so this uses ar net in the background which is a neural network that's particularly good for time series forecasting so we're able to set our number of epochs so in this case i've just set it to a thousand you might choose to train it for longer or for shorter depending on what your accuracy looks like but that's pretty much it so we create our neural profile model and then we type in m.fit and this is going to go on ahead and train our model so let's go on ahead and train it and we'll be right back all righty and that's our model done training so you can see we ended up with a final mean absolute error of about three degrees so this means that on average our model is going to be about three degrees plus or minus out or out from the actual value now that's not too bad given the fact that we trained pretty quickly and we don't exactly have a gigantic data set so in terms of performance this isn't too bad now that we've gone and trained our model so you can see that we get a whole bunch of operational metrics we actually want to go and forecast so up until now we've really just been doing pre-processing and training we haven't actually gone and forecast so let's go on ahead and now start forecasting so first up what we're going to do is we're going to make a future data frame and then we're going to use the predict method inside of our model to go ahead and forecast forward so let's do it alrighty and that's our forecast created so what we went and did is three key things so first up what we did is we created a data frame that had our forecast periods in this case we're forecasting for 900 periods and to do that we've written m.make future data frame we've passed through our existing data and this number of periods that we want to forecast this is stored inside of a variable called future so if we take a look at that you can see that we have our future data frame there then what we've gone and done next is we've used the model.predict method to go on ahead and predict our future periods so to do that what we've done is written m dot predict and we've passed in our future data frame that we wrote up here and we've stored that inside of a variable called forecast and then once we've got our predictions we've gone and viewed our top five rows using the forecast.head method so if we actually take a look at forecast.tail we'll see the last couple of periods that we've gone and forecasted and you can see that we're now getting forecasts all the way out until 2017. so remember we cut off at 2015 so this is going to forecast 900 periods after the last date that we've got if we wanted to we could forecast way further into the future so say we did a thousand two hundred and you can see we're now forecasting into 2018 so this is forecasting for 1200 days now what we can also do is we can actually plot these results out so if we wanted to we can type in m.plot and pass through our forecast and we'll get a plot so let's try that and there you go so there's our forecast plot so you can see that we've got dates as our x-axis and temperature as our y-axis and you can see that over time it's kind of mimicking that same pattern that we had up here so you can see that during the middle of the year we have lower temperatures and that's when we have winter in australia and towards the end of the year we have higher temperatures so that represents our summer now what we can also do is we can actually break down the components that represent this particular time series forecast so we can actually see the seasonality across months and across different years so let's do that oh before we do that in order to generate this plot we've written m dot plot and then we've passed through our forecast data frame and we've stored that inside of a variable called plot one to generate the component plot what we're going to do is write m dot plot components so plot underscore components and then again pass through our forecast data frame and we'll store that inside of a variable called plot2 so if we run that you can see we now have our component plots so this is going to show you the overall trend so that's that first chart that you can see there so overall it looks like that temperatures are increasing and you can see that that's represented in this time series forecast yeah so in 2015 our max that we hit was around 27 degrees in 2018 and max that we hit is probably around 30 degrees or 29 degrees there and you can see that that's represented in this trend graph here now we can also see the yearly seasonality so in this case you can see that during our summer so january you can see that we have high temperatures and then as we head towards winter we have lower temperatures in july and again it starts to ramp back up as we head towards summer and then it looks like we have a little bit of seasonality on day of the week but that's not particularly strong there as you you wouldn't expect there probably isn't too much seasonality within a particular week but that about wraps up how to actually generate these forecasts so we've now gone and used the make future data frame to go and create our future data frame and then we've used the predict function to go on ahead and predict and then you can see that you can really easily plot out what these forecasts look like using the m.plot function and m.plot components if you want to see the different components of trend now the last thing that i want to go through is quickly take a look at how we can save this model down so that we can use it later on say for example we wanted to generate a new forecast later we could do this so what we're going to be doing here is using the pickle.dump method to dump this to disk so we'll be able to pick up our model so let's go ahead and do it all right so that's our model now saved now what we've written there is with and then we've gone and created a new file called forecast model dot pico so in order to do that we've written open forecast model dot pico so this is going to be the name of the file and then to our open statement will pass through wb so this means write binary then we've stated that we want to work with this file as f so this means that we're going to be working with that as f and then to dump it down we've written pickle.dump and then m which is our model that we actually want to dump so that's our neural profit model and then we've specified that we're going to dump it out as this particular file here so forecast model dot pickle so some pretty standard pickle dumping functionality or what you typically do for a circuit learn model as well now you can see that our model is now saved so if we take a look we can see that we do in fact have forecast underscore model.pickle inside the same file that our notebook is in now if we wanted to we could actually delete our model and then reload it so in order to reload it we can pretty much copy this line here and then we just need to change a few key things so first up we're going to set instead of having right binary we want read binary and then rather than using pickle.dump we're going to use pickle.load and remove the m so what we're basically saying here is that we're going to be opening up this file as f and then we're going to use pickle.load to load f and then store it inside of m so we're basically re-initializing our model and if we take a look now we have in fact loaded up our model and just to prove to you that this works if we type in dil if we delete our model again you can see that we've now deleted it and if we reload we've now reloaded our model now if we wanted to we can go and copy our forecasting code which we had up here paste it down here and then we can go and i'd say we wanted to forecast for 12 000 days we can go and generate that forecast and again go and plot it out by copying this code and you can see we've gone and forecasted way out all the way until 2049. so that about wraps it up so we've gone and done quite a fair bit so we've imported our dependencies as well as installing your profit we pre-processed our data as well as plotting it out while we were doing that we then went and trained our model using the neural profit library and using the fit method and then last but not least is we went and generated some forecasts we saved our model to disk reloaded it and then went and generated a new forecast and that about wraps it up thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell so you get notified of when i release future videos and let me know what other types of things you'd like to forecast using your profit thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 5,401

Rating: 4.984436 out of 5

Keywords: neural prophet python, neural prophet tutorial, neural prophet install, neural prophet, ar net, ar network, forecasting weather

Id: mgX0Iz4q0bE

Length: 23min 46sec (1426 seconds)

Published: Wed Jan 13 2021