Forecasting Demand, Finding Sales Data - Facebook Prophet, Google Trends & Python

hello friends let's talk about forecasting demand this is as old as money as old as Commerce you know in rainy seasons of stores would buy umbrellas for before the winter stores will stock up on winter coats etc etc right it's also critical today in this this era of kuba 19 there's a virus that has closed all sorts of stores all around the world everybody wants to get that online presence so they can compete against the big ones that Alibaba is the the Walmart's the Amazons so it makes sense that you want to have an online presence and a great tool to help you get there is to learn how to do some forecasting and I'm gonna introduce a cool tool here FB profit is an open-source tool used by Facebook produced by Facebook and released in the public domain that is extremely extremely powerful and so simple to use is the most simplest tool I've ever used for forecasting I'm gonna walk you through a simple exercise using kaggle data actual competition data but it's a real data and then I'll show you how you can get your own your own product data even if you don't own the data meaning how can if you want if you're thinking about introducing a product but you don't have the data it's recommended you should have like two to three years if not more historical data so the model can capture the seasons the holiday is the sales the the dead times you won't have any years as you as you can have so I'll show you ways of finding that data that is a proxy to what your product you may be interested in order to be able to do forecasting on products either never seen or you don't have the data so very important to do it if you wanna learn more about what's going on so welcome to realm L show my name is Manuel I'm gonna have to get you can find me here at viral male comm V IR al ml calm please sign up for newsletter middle of page you'll get updates on my videos my deals and also my free classes there is a sale going on right now on all the classes I'll talk about this at the end of the show the there is code of course as usual the link to the interpreter extract is in the description of the video we have everything you need to follow along of what I did or to do it after after you watch the video either way so um to properly you know forecast data like I said you need to capture a lot of cycles a lot of seasonal cycles holidays sale trends so you need a few years of data and that's what we're gonna do with cargo we're gonna be using a great data set called store item dem demand forecasting challenged and they give you quite a few years right here of great sales data that we can use is anonymize we don't we know what it is and that's what afterwards we'll do the proxy we're well know exactly what we're dealing with but a quick word I found this really cool chart the link is in the description as well about visual capitalists and stack land the data is from stack land stack line I think it's called a stack line right here they've been analyzing what's popular and items on I think I think on Amazon and you see clearly this desire what was trending today in March last month in March we see disposable gloves bread machine Kaufman Kaufman right the three top products right now while they're comparing it to last year it was luggage briefcase and cameras right all these things has to go outsourced and explore it right now with covert nineteen nobody's exploring we're all stuck at home like I'm stuck at home here too so it's very interesting right these are the kind of things that you can tree for cast discs but there are also great tools stock line being one of them Amazon bestsellers that is a phenomenal resource and I think I put a link if you're not familiar with this I put a link here somewhere let me just pull it up because it's just an incredible tool we're right here Amazon bestsellers will tell you what people like and don't like it's not just for books is for any products and so if you're not sure you want to introduce a product or not sure if it there's a demand go to Amazon bestsellers gonna quickly give you a feel if it's selling or not and we'll also look at Google Trends another great tool so let's get code let's get coding here let me read you first because I'm gonna be we're gonna be using the open-source tool FB profit it's a phenomenal tool I want to read you the the kind of the the intro to what it is so we're on the same page profit is a procedure for forecasting time series data based on additive model where nonlinear trends are fit with yearly weekly and daily seasonality plus holiday effects it works best with time series that have strong seasonal effects and several seasons of historic data profit is a robust is robust to missing data and shifts in trend and typically handles outlier as well so remember it works best with time series and you should have a few years of data that's what it recommends and it's actually used all over Facebook for to put for producing reliable forecasts for planning and goal setting we found it to perform better than any other approach in the majority of cases so I know I usually I would work with tools like a ream on all these things where you have to kind of hand pick the seasonality yourself this does it all for you automatically there you can go very deep with it but we're just looking doing I like the automatic stuff and it has plenty of it here so let's get going I'm going to load a few libraries including the FB prophet here you have to import profit this was a bit of a hassle to install for me so I put the docs here if you're on a Mac it was a little bit troublesome I think you had to have Shia fan or some some compiling going on I had to revert to installing it through anaconda I don't like to use anaconda I mean I love anaconda great product I don't like to have all this extra stuff on my machine I like to keep it simple clean but I had to do it that was the only way I could do it I think is easier on on on Windows I think though I haven't tried on Windows the data we're gonna be using I mentioned here it's called the store I am demand forecasting challenge great notebooks where they have used profit to win competitions and I'll show you how they're using that data and the data that's what you need to download if you want to follow along the link is in the in the code so I really am there's a bunch of CSVs I'm only gonna use a training so here you have to read it this is exactly how they read them you know copy tally do it in the notebooks low memory false parse date you know and the index column so I haven't thought about that much I just want to get data quickly to defeat it into profit so we're gonna sort data by ascending order and we are going to look at the shape and the head of the data we see it's anonymized so we all we have is a store that's about 10 of them items there's a whole bunch of them and the sale you know basically did a sell and that date very important right this is time series data so very simple just three columns of data with the index or four if you count the date as well so let's let's just plot this stuff so very simply I'm going to look at all the sales for store number one so we're going to take a subset of the data we're going to say give me all the store that are equal to one basically the first store and plot all the all the items for that for that store so there's a whole bunch of them right I think there are 50 items so you're gonna see all sorts of you know lines here it's actually pretty looking and you clearly see there's a cyclical effect it looks like it's sales that most sales in the summer maybe they're selling something that's a summer like you know swimwear no I didn't look at the details of what it's supposed to be now let's do the flip of this let's do all all the item one so whatever product that is for all the stores so here we're gonna see ten stores and each one of their items right yeah I'd item one for those ten stores I don't even know they're the same item I'm gonna assume they are I was very simple to do right I'm doing a loop just in case you're interested here I'm doing a loop over the subset of the data we're here I'm saying give me all the stores for I think the have item one and create a set of it so we're gonna have a unique set of those stores and we're gonna loop through store by store and in those stores we're gonna say you know give me for that store right give me item number one right for that store give me the sales and I'm on top of it I'm going to be adding a rolling average so kind of smooth it by a 50 period because otherwise it's really you know Wiggly and you can play around with that but it gives you you know kind of quick clear indication that you can see there is seasonal effect and we know that Facebook profit likes seasonal data right there's not something to use on the erratic stock market data is probably not gonna help you out but on with something seasonal you're gonna learn a lot especially it's gonna prick into the future that's what that's the power of the profit so we're gonna bring it in to keep this simple if a profit is is aspheric it's a fairly a big big model so if you have a lot of data it can be very slow that's why I'm gonna take only item number one for store number one keep it simple I'm gonna reset the index and I am going to rename the columns to D s and y so d s for date and Y for whatever your you know your outcome variable is we do this because that's what the default values that Facebook profit expects and I bet you there's ways of changing that I didn't want to deal with it I just wanted to see the results I'm also taking a sample of it making sure I don't have too much data you instantiate profit right you just call whatever variable you want to name it in my case is profit underscore basic equals profit right open parentheses close parentheses you instantiate your object and you pass the data in the correctly formatted way right it has to be two columns with these names and you call the dot fit and that's gonna take you know a few seconds and it's going to model the data right but that's fit is just like you would with any other machine learning tool now we have it you're gonna create future dates in this case I'm gonna say take the data frame and add 365 days to it right because that's what I want to predict simply you say you pass it your data frame you say make future data frame period 365 and that's it right so so so let's look at this in more clearly right so this is babe profit on score basic is my fit model for that data item number one for store number one and you call make future data frame a function and you pass it how many days you won't know how many periods it could be weeks right depending on what the granularity of the data is in this case is daily data so I'm gonna say give me an extra 365 days or one year and let's look what it looks like the whole data so it starts as it did in 2013 we can confirm right it starts in 2013 and now it's going all the way at the end of 2018 and if you look at the data it only ends in 2018 so it added a whole year right basically all of 2018 which this one reads at the beginning of 2018 so now we have an extra year of blank values and it is when you call the profit underscore basic our training model and you called a dot predict function so it works just like any other machine learning and you pass it your your new data set which is future which includes the future dates I'm running it takes a little bit of well and for the plotting so we also now that we have predicted the values we're gonna call the dot plot with our forecasted data and look at that it gives you these beautiful plots you clearly see right the cyclical nature of the data the black dots are the actual data points and here you have the the predicted future clearly there's a little bit of a linear it's a positive linear slope slight and you clearly see it keeps on going up so this is the future we predicted in the future demand for whatever product item number one is it gives you more than this you can do a plot underscore components and it's gonna give you interesting data where are you here it's gonna give us the overall trend over the years right from 2013 including our year we're predicting 2018 2019 it's gonna tell you that these are the weakness is pretty cool when it's popular apparently Monday is not popular maybe the store is closed on Mondays and the weekend seems to be the most popular and of course the of the year and that we knew right is that this they probably selling some summer item and it just jumps up in this summer great demand you can also do what they called the change points to the plot and it shows you highlights where the data is changing so you can you know can I dig and understand what when they're changing the changing nature of your of your sale of your item through time and and also you know it kind of gives you a nice linear the linear slope right here so that's what it does so what do you do if you don't have you know years of data with you there are there ways right there's Amazon bestseller is that that stack line chart I showed you which is pretty close telling you what's I've been properly was not properly but a great way of doing it is through Google Trends I'm gonna show you what I just did this exercise and we're gonna we're gonna redo it together uncle trends for normal tool I love it I go there all the time I'm gonna do a search on the vitamin D we know that now with a covert nineteen everybody is becoming health nuts we all want to make sure we have all the rights vitamins we have all the equipment we need to survive so I'm going to do we want to get a few years so let's do five years that's great on this data and I'm gonna close this and here we see right we can see this seems to be a cyclical nature vitamin D you know people really like vitamin D is kind of a substitute for sunshine so in the winter there's gonna be a demand for vitamin D and that's what I can't really see what the dates are but it looks like it's winter we have peaks so this is exactly the kind of data we want to analyze I'm simply gonna hit the download button and it's gonna download it to my machine right yes here it is so you may not see all of it but this is the data it gives so remember we want to change a few things instead of week put D s and instead of vitamin put Y right that's just gonna that's just the way Facebook's likes it and remove these two columns and now we certainly without doing any work we have everything in perfect format for to run this through FB profit save this and you're good to go right save it as a CSV and now we're going to jump back to to our data I am going to bring this data I called mine vitamin on this vitamin underscore D dot CSV Yuma Lotus in memory I am going to create the date/time make it basically a make sure it is a date time because I don't want to sort it as date time in ascending order and we're gonna see the shape and we're gonna do the tail of the data see make sure that we have fresh data and look at that yeah this is the basically last day of March when I ran this we have the demand or the interest on Google Trends for this product so I'm using that as a proxy right remember this is a percentage of the viewership but it's gonna work great to you know to figure out as a proxy to demand if you let's say but let's imagine you wanted to sell vitamin D this is what this is one way of doing it if you don't have the data or if no other your competition doesn't want to share the data you can use this so now we have the data right we have our Y column which represents the interest in vitamin D okay so we now have our data we're ready to go let's feed it through profit so I'm gonna take a copy I'm gonna make a copy of the data set I'm going to instantiate profit just like we did before and I'm gonna pass that data set to profit by calling the dolphin method and I'm gonna add an extra 365 days even though the data is weekly we're gonna add it in days FB profit will do all that for us in the background and let's look at make sure we have a year's worth of data and yeah it takes us right around end of March 2021 perfect we have a year's worth of data so now we are going to call the predict so we have our train hour we have our train model it's called future so I know training model is called Project profit basic and we have our data it's called basic it's called future so now we're going to forecast we're going to call the predict method so you our training model is profit underscore basics dot predict and we pass it our future data that's what I meant to say and we have our forecast and then we're gonna plot it let's plot it to see what it looks like and here it is look at that right so we clearly see we have data starting 2015 and we see we have this peak of interest in vitamin D in it looks like the winter month right January February we're gonna confirm that because it's gonna tell us later on and we also see this is very interesting this peak demand right this is Cove 819 we can confirm that in in in Google right it look at that in Google Trends is going up is the highest point it's been in the past five years or however many years we have here a lot of interest and that's what people are you know very interested in being healthy with you know with covert 19 and it clearly shows us how it's projecting the interest is gonna continue as a meaning that there has no idea about covert 19 but it's just looking at the overall trend is definitely a positive linear slope high demand is predicting the continuation so if you wanted a stock right if you wanted to stock or vitamins you have to figure out how many to do and then kind of assume what are the numbers that we need to looking at these charts right obviously you know middle of a month right you the middle of the year less interest and the end of the year beginning a new year huge spike in interest for vitamin D we can also look at a you know this is the the components also very interesting where are you here we see right there was you know somewhat so-so interest that just spiked in 2020 and I was predicting from season 21 a huge interest in vitamin D and here we confirm this is very a lot of interest during you know the winter months January January March a lot of interest and any goes down July you know the interest wanes when the Sun comes out people don't seem to want vitamin D because they can get it directly from the Sun so that's what I wanted to show you right this is how you predict this I love Google Trends that you can use it for so many different ways you can slice and dice that data for you know all sorts of forecast also predictions for all sorts of any information you want to know about what as in aggregate the whole world is thinking well it's a great tool to also look at you know if you want to you interested in store data demand forecasting demand you can use it because what's really cool is it allows you to drill down by different countries different timeframes different categories you can even you know web search will be like for Google search YouTube search depending crisises a lot of ways of slicing and dicing this data and at the end it always allows you to get a CSV which then you can do modeling you can do a forecasting with FP profit etc etc I love it so before I end I like to plug my material quick word on the viral ml school this is 50% deal going on right now we're all stuck at home I want to make sure people get to study hit the classes button at Varley milk calm and you'll see my different tracks here everything is 50% so you choose whatever you want or you can go with it all track which is everything the closest I think interest to people here would be the fundamental market analysis where we do look at analyzing the markets analyzing what's going on fundamentally in the world interest rates unemployment you know PMI CLI real estate all these things are very important and python is a great tool to you know to explore that data and it's also the lot of machine learning ones including my latest crack is called the ML web makers where the goal there is i'll show you four different ways of porting machine learning to the web so front-end web models which run directly in the browser of the client back-end where you run it on a web server pre-trained where you train it elsewhere and then mount it in the model and also REST API and this is great for people who are thinking about building an online presence so if you want to sell something online if you want to build a store or even just you know have a blog these are great tools to kind of make make your site intelligent machine learning intelligent great ways of basically expanding at taking your you Jupiter notebook Python work into the web so the entire world can you know can use it not just people who have access to github thanks for watching [Music] [Music]

Channel: Manuel Amunategui

Views: 10,972

Rating: 4.9742765 out of 5

Keywords: FBProphet, Python, Google Trends, Forecasting, machine learning, deep learning, data science, time series prediction, time series forecasting, time series analysis, machine learning basics, data science project, data science career, time series analysis and forecasting, facebook prophet python tutorial, facebook prophet time series forecasting, python tutorial, fbprophet forecasting, google trends tutorial, google trends 2020, Ecommerce, ecommerce business for beginners

Id: AX1wKnBPhvU

Length: 19min 14sec (1154 seconds)

Published: Sat Apr 11 2020