181 - Multivariate time series forecasting using LSTM

hey guys this is trini and i'm recording this video based upon your request in fact in the past we have covered the topics of lstm what they are and how to use it for a time series and in that video i've used a single variable meaning autocorrelated time series right you have only one data and you're trying to forecast based on that one data okay that can be stock price or in in fact in the past we looked at airline passengers how the passengers were you know the number of passengers actually change uh or increase or decrease with respect to time and how we can forecast that so uh here is an example where we'll do exactly that so not many tricks actually if you understand that then they should be able you should be able to understand this in an easy way except instead of one variable we'll have multiple variables that's pretty much it okay now uh let's jump into it and as usual i'm going to use python and keras but one thing i'd like to highlight is again watch videos 165 and 166 because they provide some introduction in case you already don't know what lstm is and how to use it for time series and for today's video obviously i need some sort of a multivariate data set and the best thing i thought would be stock market uh of course if you have other data sets please go ahead and use it this is the best one i can think of especially because we used exactly the same data in the previous video right now the video before this one where we did auto encoders using lstm auto encoders and we looked at reconstruction error and looked at anomalies in in the data so this is exactly the same data set except instead of only looking at the closing price or opening price let's go to historical data so it's going to show us a whole bunch of data so for each date we'll have the opening price for the stock mark for the stock here high low and close and adj close i have to look down exactly what it means and also the volume i don't think we'll be using volume even though you can use it because we'll be normalizing the data sets so what the plan for today is use these one two three four five columns okay as variables including the first one okay all of these as variables to predict the opening price we can also predict the closing price it doesn't matter let's predict opening price using all of these as variables so we have five variables and one thing one of those is some uh what we are trying to predict and the assumption is okay so these are the right variables that are affecting the price you know for the open price okay so with that knowledge let's just go ahead and jump into our spider ide and i have tested this on tensorflow 2.2 and keras 2.4.3 but there is no reason it doesn't work on other versions okay now let's go down and import our libraries these are pretty much the standard ones no tricks here we are importing lstm and dense layers because these are the ones we are going to use and using keras models sequential method we are going to put together our lstm that's the easy part the tough part is getting the data ready in the right format once you have it then this is pretty much straightforward okay so let's use pandas to read the csv file and let me switch to variable explorer remove all variables here from my previous runs i was testing something else and let's run these lines one more time okay so now it's very clean so our data frame should have seven columns one for date the five variables we talked about and the volume okay number of transactions that day okay so what shall we do now so uh if you look at this the date is an object you can test it out this is like text we need to convert that into date time format so i'm gonna convert that into date time so only for plotting purposes otherwise it's useless because our five variables date is not one of the variables right so what i'm trying to do here is extract the training dates into a separate series so you can see the training date right here so later on when we plot we can use that as our x-axis that's pretty much it okay so that's the reason i extracted that and now let's extract the columns that we would like to use as variables so these are the columns number one through six in our data frame right so column number zero is our date and one through six is uh one two three four five six actually including the volume okay so let's go ahead and include these of course you can go one through five if you don't want to include the include the volume but let's just include it for now so let's go ahead and do that and now for these columns i'm just creating another data frame and again you can see oh this is excluding six so i'm not including the uh the volume i always forget that anyway uh so our new data frame is basically the old data frame with these columns without the date and the the volume right okay so this is our new data frame and i converted all the values to float so when we actually do normalization you don't lose any information there you can plot it i was doing some experimentation they aren't all i'm trying to do is okay let's look at last 5000 data points and see how they look like but it looks very boring so let's go ahead and comment this part out the next step comes uh a scalar okay so again you know why we are doing this because let's go back to our data frame you can see the values look like they're all very similar but if you actually keep going down well in this case they are similar because they're all the closing and opening so they're very similar but scaling them is always a good idea if you especially include the volume column yeah where you have those in hundreds of thousands because that many transactions happen uh in a day then scaling the values is even more important so all i'm trying to do is use the standard scalar and scale all of these and you have to keep in mind one thing when i do standard scalar i instantiate this and then i'm fitting this scalar using the data which is available as part of df for training so if i open this data frame for training okay it's going to use all one two three four five columns to do this i could have picked only one column the reason i'm stressing a bit on this is later on when you want to do inverse scaling because once we scale this the values are between i don't know minus 1 to 1 or minus 2 to 2 depending on how our data is so this is our data set but then later on if you want to get the original data back after the predictions okay the inverse transform expects the input to be five columns because we are using five columns here sorry for dwelling too much time but i'm pretty sure you'll run into some issue if you are not familiar with this and then it takes a while for you to figure out why it's going on so if you want to fit it you can actually because all of our data is within the same range in this example we we could have actually fit this only on one column and then applied it to the uh transform everything okay just a quick note but i'll show you another trick to uh while we're doing inverse transform okay there you go that deserves one minute believe me okay so now this is the meat of multivariate because the lstm part it doesn't care it just looks at your input shape and then it does the training but how do you get the input shape ready uh it's again we have done this in uh almost all of our time series based you know videos so all we're trying to do is uh you know you can write a function for this or you can just do this on the fly so what we're trying to do is capture uh our training and the prediction right so train x is our training series and train y is the prediction so what is train x and train y so this is again time series which means if i take my first value up to the 10th value as my x then the 11th value is my y it doesn't have to be 11th you can also say 10th uh you know 21st value is your y so you can look forward in future but typically it's customary to look at okay i want to use 10 as input and then the next one as output and then move this by one right 10 as input the next one has output so you can do this as a sliding window and like i said it doesn't have to be one your output can be the third one or fourth one and so on so we are going to capture that so we are going to reshape our our five variables that we have into these chunks and then what the uh you know value we are trying to uh fit it to okay so that's what end future is how how you know how far in future are you going to predict let me just put one that basically means okay go from one to ten or in this example go from 1 to 14 i want to see two weeks behind okay back and then 1 to 14 and 15 value is my prediction is the next one and then go from 2 to 15 and 16 is the one is my prediction and so on so that's exactly what i'm trying to define here and that's what this this this for loop actually does so let us run these lines okay and uh you go and now convert this into if i go back here you can see my end future where is it train x and train y these two are lists so let's convert them into array that's all i'm trying to do here and i'll open this in a second once we convert that into error it makes it a bit easy for us to see you see how my train x is these many data points okay 12 0809 it's not 12823 because we are holding 14 days or we are so uh you have to account for that okay so 12809 let's go ahead and open it by 14 by five what is this 14 the 14 is the 14 days and uh you know that we are looking back and the five is the five variables that we got okay now let me open this this is the key part everything else should be easy and let's look at df for training scale this is the entire data set this is the training data set and let's look at the y values right there okay so this is variable number one which is the open second one is something else and and one of these is close you you know what those are so we are starting with negative 0.973789 right so how far down are we going 13 0 to 13. remember total 14 points for our x value so each of this column is going into uh into the into the system you know into into our lstm as a training vector okay so point nine seven nine seven so let's go down to thirteenth one nine seven six nine four six nine seven six nine four six so what is the y value for this what is after nine seven six nine four six it's nine seven six three eight nine that's exactly what the y value is nine seven six three eight nine right so for these values we are predicting the next one and not if it is just a single variable time series it would be only this column because this is multivariate we have additional four columns that's the only difference everything else is exactly the same so for this matrix of values the prediction is 0.976389 for the next matrix let's go one index up we have 12809 such matrices with this prediction so let me just go by one that means my starting value now is let's go back instead of nine seven three seven eight nine now it is nine seven four four three nine okay and it goes all the way down to fourteenth value which is nine seven six three eight nine sorry nine seven six three eight nine and what is the prediction right there nine seven six two nine six which is this one i hope this makes sense if so you know everything you needed to uh please don't stop watching the video because maybe you learned something but basically this is it once you get your multiple variables in this format you're all set okay if it is uh if it is a univariate thing instead of the instead of this shape being whatever the number of data points and number of look back or the days that we are in uh or the sequence size uh and this would be one right because you had one variable now we have five variables that's that's the difference okay i think i dwelled on that way too much i hope it makes sense to you guys so once it's done now you can go ahead and print out what the shape is in case you cannot read it up there let's since it's here let's go ahead and do that and as you can see the shape that's going into lstm into our neural network is 12809 different x values of size 14 by 5 and the y value is 12809 we have size 1 because it's just predicting the day after okay how do you put your model together exactly the way we have done every other model so in this case we are doing lstm with 64 units and my input shape is 3x shape 1 and train x shape 2 right so my trainex shape 1 and 2 is 14 by 5 right there and this is important return sequence is equal to true why because i want this lstm to return a sequence for the next lstm to process that's why okay because i have a stacked lstm if you if you are not including this then change that to false okay otherwise you'll see some errors and return sequences for the next one is false because the next layer is just a dropout layer or you can directly go to dense layer it doesn't matter but if you say return sequence is close to false it's not returning the sequence it's returning a vector that goes into your dense layer and then you're compiling it and you're looking at the summary that's it okay so let's go ahead and print the summary a dropout is not defined did i remove it layers let's go ahead and include it i don't know why i did that sorry about that so let's include dropout okay so now let's come back and run this one more time there you go so okay here is our structure 30 000 parameters that's fine it should be pretty quick okay even on my system but i'll pause this in a second so now i'm doing no tricks again just model.fit train xtrain y exactly the same that we normally do let's only do 10 epochs for now okay let's run this and i'll pause this and i'll continue in a second okay so there you go it's done after a few seconds so let's go ahead and i commented this for some reason but let's go ahead and uncomment it and have a quick look at how the training went um okay i guess that's okay let's uh now go ahead and forecast it now how do we do that i defined this in future again i know i used it here and i defined it like as one i could have used a different uh variable when you are writing your code change it to a different variable because i just don't want to forecast one more i want to forecast like 90 days like three months and then just plot it so we can have a quick look at it okay so that's what i'm trying to do here so just my future days number of days is 90 and then all i'm trying to do is uh extract the dates okay what would be my future dates because when i want to plot i want to plot plot the forecasting along with the original right so i need to extract the right dates so the way i'm doing that is okay take the current date minus one okay so that day and then uh go to 90 days in future with a frequency of one day okay so that's exactly what this does and if you look at the forecast period dates it's just a list of all the dates starting with october 29th which apparently was the last day where the training stopped and then we are going all the way to i don't know december or 2021 january 26th if we all don't die by then of course virus so sorry for the grim note over there but uh anyway let's get back to a more positive tone so uh forecast what's my forecast it's just model.predict okay train x from uh uh what is my in future uh number of uh it's just one okay so let's go ahead and forecast right there and you'll see that our forecast is of the size 90 by one okay so we are just having a single column right there so that's my forecast okay now here comes the trick so if i look at the forecast it's all again in the scaled space right remember we scaled it using a standard scalar now i want the original numbers back what is the actual stock price this is not the stock price this is the normalized or scaled values i should say so for that if i just do uh if i just do inverse if i just do inverse transform for example i think this makes sense let's go ahead and do that if i just do inverse transform right i mean this is how you do it scalar dot inverse transform forecast it's going to throw an error saying okay non-broadcast output operand 91 doesn't match the broadcast shape 95 what that means is hey you only have a single column but when you scaled it you had five columns right so this is what i was talking about so uh for that i mean there are a few ways you can do i mean you could have actually gone back to original data frame and copied those or the scaled data frame copied those values for the other variables many tricks what i chose to do here is i just used np dot repeat to repeat that column how many times uh for training shape one which is nothing but uh five right so df for training df for training shape one is five okay five variables or you can just type five right here it doesn't matter so let's just do four cached copies four cached copies if i open this it's just the same thing it copied five times and then i'm gonna drop everything after inverse transform you see that's exactly what i'm doing i'm doing inverse transform but i'm only looking at the zeroth one which means i don't care about everything else it's a weird way of doing things but it works okay so now i have my why predict future so this is the predictions these are the predicted values there you go i still have one column and now you see how the values are in real dollars and not like negative zero point something so this is it this is our prediction now all we have to do is attach this to the right dates and plot this with the original data that's all we're trying to do now okay so my forecast dates for time in the forecast period dates okay go ahead and append this so we'll have that part and then we are going to capture that the forecast dates okay and uh y predict future both of this into a single data frame and uh you see how i defined this in the uh in a dictionary where my column name is date and here the column name is open so again i hope you're pretty good at pandas uh so there you go so my data frame with forecast is this and now i'm converting my date back into the date time format you may think like okay why are you doing this and you could have handled things in a better way of course we could have handled things in a better way this is something i put together uh pretty quickly and uh i i was writing stuff as i was going along so if you are a better planner you probably have could have reduced this code by a few more lines okay so now we have our data frame for forecast right here and you have the date and you have the the the opening predicted values we could have just said opening predicted and now let's do exactly the same with our original data frame right so what is our original data our original data is in df right there this is our original data and we just need to extract the date and opening time and let's go ahead and do all of these okay so there you go all i'm trying to do is basically get original data from the original data frame okay and only the two columns date and open this is what we need that's it we don't need anything else and here i'm just looking at everything that's after may 1st 2020. otherwise the forecast is that tiny and then the entire graph and it's it's tough to see things so this is only for visualization purposes okay now let's go ahead and plot it again i'm using seaborn you can use pi plot for plotting so here it is let's see how good it is oh it's not bad i mean i could have actually overlapped some of these and you can see how the prediction actually goes i'll leave that exercise to you in fact you can predict all the way from the beginning to here and then see how it continues but anyway so here is the original data and here is the forecast for uh i don't know 60 days uh how many uh future 90 days in future okay so i hope now you know how to work with multivariate data set just like linear regression multi-linear regression nothing it's pretty much the same thing you just get the data in the right format same thing here linear uh you know or sorry your univariate or multivariate you just get the data in the right format then you're all set okay so thank you very much for watching this video and again keep requesting other stuff and if i have time and if i have skills in that field i'll definitely make a video on that topic or i'll do my research online and i'll digest the information and regurgitate to you in digestible chunks so you can understand it but please do leave feedback in terms of if you want something you know that you're struggling with and if you think i i can help you out thanks again please do subscribe

Channel: DigitalSreeni

Views: 255,031

Keywords: microscopy, python, image processing

Id: tepxdcepTbY

Length: 22min 40sec (1360 seconds)

Published: Tue Dec 08 2020