Igor Gotlibovych: Deep Learning and Time Series Forecasting for Smarter Energy | PyData London 2019

while someone promised quite a lot there thank you all for coming I know you have a choice of talks to go to so thank you very much and also thanks for sticking I know you're sticking with my data for two full days of talks it will not be too mathematical hopefully for Sunday afternoon so I went on the internet this week and I found this now I'm not going to be talking about this type AI for energy I'm going to actually be talking about fairly implementation and we're going to talk about implementation I'm going to be talking about as a specific tools that we use so this is not going to be a marketing slide about always marketing debt about how we are going to revolutionize everything with AI now why is machine learning important in particular in this energy industry well this is one of the reasons so this plot shows every day since 2012 and it shows a percentage of the UK energy as it was generated from burning coal so we can see some very positive developments in fact and in I don't know if you have been following the news more recently we've had some prolonged stretches of extremely low fraction of coal on the UK grid in part due to good weather in part due to sort of policies and politics and various shifts in the landscape but that comes with its challenges and I'll try to outline some of them very quickly so the traditional energy grid it basically has the supply was a generation side we have some big power plants you turn them on their run for months they've done a lot of whatever it is then there are some intermediaries there are some markets and if you're a domestic customer if your household you probably have heard of the big six you may have had your energy delivered by iana British Gas for decades and the household is sort of it just sits there for 20 years and consume some energy now things are changing things are changing across the spectrum things are changing as a generation side we now have far less predictable generation from renewables there weather-dependent we much less centralized and more distributed generations a solar plants tend to just be spread out in lots of little patches we have changes from the domestic side customers are getting smarter homes they're getting a battery-powered vehicles that place new demands on the grid but they also installing their own generation right so octopus energy as a consumer facing retailer we said somewhere in the middle now of course we love doing things better than the big six and we love doing it with data and technology and we have a lot of projects across the spectrum but one of the key reasons why we're in a position to do what I'll be talking about is quite simple we know where our data is and actually it's surprising how many companies don't so given that we know where our data is we can make graphs like this right we can make graphs like this so here is a particular week from some months ago where there was there were several dates in green you can see the green energy generation this is what 10 gigawatts of wind on the grid and overlaid on top of the Green Line which shows the wind generation is the price signal so this is where the only retailers that offer a fully dynamic price signal to our customers if they choose to as there are people who are mining Bitcoin when the price dips it's an early adopter thing at the moment but we can we can do this right so we can pass this pricing on to the customer and the price signal can go negative now the price signals a reflection of the imbalance between supply and demand and the markets and the grids efforts to balance it out because electricity has to keep flowing you can't just store it very easily right which is one of the challenges and then we can look at our customers and we can look at the smart meter data that we have been gathering over the last year and a half maybe and we can do various things we can see we can plot all the customers who are on a fixed tariffs versus all the customers who are exposed to this dynamic pricing and we can see a completely different consumption behavior now this is something the National or many other companies do not see because they don't have this data right we can look at it bottom up so we can see this massive change the blue is more representative of what the industry thinks everyone looks like but we know that each customer has a different consumption profile and in fact many of these people have a massive spike at night because I know the price is cheaper we can see when they're charging electric vehicles and we can see that now of course we have helped this a little bit by sending a few notifications and press releases because negative pricing while it's just another number right it does generate some headlines so people suddenly charge about twice as many EVs and you know they save a few pennies and we do genuinely pay them I mean so you have a unit rate usually you pay I don't know 10 people kilowatt hour right now the price owns and grid it fluctuates all the time depending on supply and demand if there is a huge oversupply it's really windy the market can starts pushing the price down there is an auction happening and basically that price can be pushed negative when you get paid to take off the energy because it's too expensive to dump energy it's very difficult you can't you know you can pump water uphill you can do various things but yeah so and we pass it on to consumer to consumers on this particular tariff okay so this is the sort of data that we're dealing with and we can disaggregate it further we can look at customer level and then we start seeing this now we have tens and tens of thousands of these time series and one of the key challenges that will enable us or already enables us to do better to make better trading decisions to make better hedging decisions but also feeding into work like optimizing batteries for the day ahead to better make use of these imbalances is being able to forecast this now as you can imagine it's quite noisy and you know this is the sort of challenge that we are facing by the way in case anyone was looking at this IDs and singing of the German tank problem and how many customers we actually have it's actually quite public we are very open about how customers we have so we have about 77,000 smart meters and this about 10% of our total customer base so and that red line at some point that the red line was going up at such a slope that I was worried if we can scale quickly enough but you know luckily it flattened off a little bit so the rest of the talk is less about energy and more about sort of how we approach timeseriesforecasting and I think timeseriesforecasting has a bit of a reputation within the data science community and it's it's often considered difficult it's often considered a separate field and I think some of the main reasons is that there are so many approaches and this is compounded by the fact that there are so many approach is emerging from different communities different topical communities different fields of machine learning and mathematics and this makes it certainly for me when I started it made it very difficult to know where to start you're kind of torn between so many different approaches so try to demystify this a little bit and of course bias it towards how we do it but without pretending this is the only way right so fundamentally what we are dealing with like was any machine learning problem we have some features and we have some targets we are predicting a value we want to know how much energy is this household going to need you know on this date at this time and then we can sit down and think about what our features are but the first thing we need to decide is at the regression problem yes basically timeseriesforecasting is a regression problem and that kind of points you to the right half of scikit-learn so and then you can ask yourself what are the features obviously time itself is a feature now pandas has date/time as a type and lots of other libraries do scikit-learn does not like date/time as an input type so this is something we need to think about but other than that it's just a regression problem now the final point that I would get to in this talk is as you saw the behavior is quite unpredictable in our case we expect to be inaccurate and I'll talk about what this means so we should start worrying about predicting some probability distributions which we do routinely in classification we say this is a 70% likely to be a cat but with regression we often predict a single number so I'll talk about that right so I said we have the features and we have the target and then people come and say ok but I read about all these AR models and ARIMA models and a gazillion other models I mean yes and the reality is you can always draw the sort of diagram where YT is the point you're trying to predict and so this is in the sort of loosely applied plate notation so for each time step you you just draw the arrows what does it depend on well in the simplest case it just depends on some features that themselves depend on time in some more complicated totally aggressive case the previous value is included with the features right so it may not be computationally efficient to treat it in the same way as there's a features but ultimately that's what it is why depends on these all of those features including previous values sometimes we want to use latent variable models which is something the sort of PGM the probabilistic graphical model community is a Stan community is very familiar with so there we want to have some variables per consumer so think of these squares as four loops right to the inner for loop is over time and the outer for loop is over customers or households so we could have a latent variable per customer that somehow effects the outcome or even external features per customer rather than per time step and we can draw these diagrams in any shape now another popular approach to Auto regressive models is to first aggregate the previous data right using some form of encoder and then we use that encoding as an input feature to the next step by mixing it with the other features so it's a bit simplistic but it really captures every possible time series forecasting model so by going to general sort of maybe it doesn't really tell you how to implement it right so people look at these arrows and these models and some people will say you know we can use stand we can use base in modeling other people will use their favorite technique right now we went down the deep learning route and one of the key reasons and I think this is one of the main sort of success stories of deep learning I mean it's very powerful in tasks that were impossible before but it scales extremely well all the sort of Markov chain Monte Carlo methods they don't scale nearly well enough and remember we have where we have sort of approaching a billion data points which I know sort of people from bigger companies probably to laugh at but we also have hundreds of thousands of time serious as we forecast simultaneously and we only expect this to grow so we went down the deep learning route and in fact what has been done recently with language modeling with all the other approaches it gives us a sort of dictionary how do we translate terms from other fields into deep learning so basically if you're familiar with sort of a are models and you have excellent and the features you see translated into sort of features as the eggs of features the features obviously anything a are becomes an encoding of your time series latent variables become embeddings and I'll talk a lot about embeddings and then anything that has to do with sort of signal processing filtering smoothing it can all be done with convolutional and recurrent networks right so you know you pitch that to your colleagues and if you work with guys like Jamie says sure let's do it so okay so this is the sort of thing that we are looking to implement right and we have the choice we have a lot of choices we have some data and features we have some household features which we can feed to an embed uh and I know there are several talks mentioning embeddings sort of this weekend and we can also look at the most recent raw data itself and we can pass it through an encoder and now this is where it gets tricky those of you who tried forecasting stock prices with the less DM so that like no it's actually very easy to get the next step ahead seemingly very accurate and very hard to do to go beyond that as I will not talk much about the encoders I'll just say sort of when we do use it it's mostly not worth the computation time in terms of improvements we get and also we need to sort of pre-trained it was an encoder decoder architecture cell not go into that in this talk okay so let's start with the input features now here I debated a little bit whether I should show codes it uses a library that no one else has so we just open sourced it and maybe some people will find it useful it's basically allows us to feed data from pandas dataframe through scikit-learn pipelines and into chaos models with multiple-input multiple-output so I'll just leave it at that but other than that the code should be easily recognizable we take the date/time we experimented with different ways of encoding date/time we generate some extra features like you know day of week they're a fractional hour this is a bit simplified and then we encode those features in different ways so and we end up with something like this so maybe for the time of day we we like using periodic features so we use the sort of sine and cosine feature which is a very useful trick by the way for day of the week we use sort of one hot feature so you can see is that Monday is a Tuesday is it Wednesday or whatever right and then we have some seasonal features across the duration of the year and then we would add any external features from the weather and such like ok so then we need to build a neural network that at the moment exists in Karis world but we would like to plummet to those features now this is where it gets a bit interesting so I showed you this diagram with multiple parts of the model and we would like to be able to access the encoder we would like to be accessing better we would like to be able to access a combined model in various scenarios so the way we handle it we define several sort of karats models that share the computation graphs as I share the layers potentially the inputs and then we just you know all we need is a dictionary of the shared model so we can address one or the other and call predicts on each one of them so as you can see embedding there is an embedding layer in chaos who here you scare us actually most people yes this should be very familiar right this is a very simple model so we take the embedding we take the external features concatenate feed it to some dense layers actually I don't know how many I left one dense layer here yeah ok so a few more layers but that sort of thing yeah it looks like this right now because we defined it as a dictionary here with some convenience methods we can feed in just a household ID and get out the embedding by just calling this model dot predict on this model all we can feed it both inputs and get a sort of overall prediction we can compile each sub model and so on right then we do the plumbing as we take those feature pre-processing pipelines it could be daytime pipelines encoders it could be anything and we just say which inputs of which models are are in better model has one input and it has an output but we don't really care and then for the forecaster model it has two inputs in this case and one output and I feel like most of the scientists have written some kind of pandas column select a pandas value select a transformer this sort of thing and scikit-learn is actually getting better but I feel it's not quite there yet in terms of feeding pandas directly into models right and then we end up with this object so this object that we use it allows us to fit the original model so it provides a familiar API and it has the extra keyword argument which is just a model name right but the details are not important basically we can train the full model including the embedder on the whole data set we can batch it the way we want we can use batch generators which is particularly important when we do actual sequence encoding because we can't preach ener 8 all the patches in memory because one time series can yield any number of overlapping windows as we do it sort of online during the training we can then do things like we can freeze everything except the embedder which is particularly useful if we train a big model but then you add an extra customer right and this is where embeddings have a huge advantage of let's say one hot encoding right we don't really want to encourage one hundred thousand customers one hot butt represented assigning a fixed-length embedding gives us the representations the customers it's easy to Train easy to update and actually gives us some neat features further down the line that we can use in other models now by having this encapsulated models it's a bit on the production side we one of the reasons to encapsulate all this in a single model object that includes the feature pre-processing that includes a sort of simple dot fit that predict API is to make it easier to run in production and the main reason for that is because we have a very small team and we have no one to hand it off to so we can't just say okay here's a notebook so having this model that has a dot fit that project and we put it you know we put our framework and all the dependencies in a docker container and we can run it we can schedule it we can run it all through this we can run it monthly we can run it weekly and we can do different updating so here I'm showing how new data is coming in new customers are getting edit right and every time we do updating we only update the final layers in this case this is a representative of just the embedding part so we update the model this embedding themselves will evolve over time and maybe someone comes up with a new model we deploy a new model and it starts running every months it does a big training run every day we use it to generate some forecasts for traders okay so we do all that so what does it look like well it doesn't look too great you can hopefully see that those customers are randomly selected except this one this is our CTO those models are kind of capturing the behavior but it's actually very difficult to see how well they're doing now of course we're doing cross-validation and we're looking at error metrics which gives us sort of a good indication of how the models compared to each other but you can probably anticipate that if one of these spikes is someone putting on a cattle it's very difficult to predict and we are not claiming to be able to do that it looks a lot better if we aggregate it if we aggregate the large set of customers as you would expect and a lot of the time for trading decisions is what we care about and if we look at the individual customers and we do some aggregation what the model clearly does learn if we are great enough it learns that this customer mostly has a higher consumption in the evening hours this customer has a higher consumption in the morning hours and this customer has a higher consumption at night time compared to the other customers so the model and then you can start tuning your hyper parameters and comparing different models and all that so we can look at the embedding space I mean I don't know if it creates the illusion of being three-dimensional but it is sort of three dimensional different people see different things in this space long enough now I you can color it by different things you can color it by the sort of total consumption yeah you can color it by various ratios you can color it but the important thing is it groups similar customers in the similar area of the latent space which is something that has been demonstrated was wet of his word vectors time and time again we obviously didn't invent this approach but it's very useful in work so well in so many areas right and we can we can then do nearest neighbor in this space if we want some clustering in categories of customers right now how do we deal with the errors and you may notice I'm noticing that I'm running out of time very soon so we look at the errors and we find that our errors themselves are actually quite badly behaved as they are not Gaussian they sometimes have sort of double peaks they have fat tails horrible so how do we deal with that well the first thing we can ask ourselves can we actually improve the model is it the model that is to blame there is no definitive answer we can try we can add more features we can do cross-validation and hyper parameter optimization but in practice there is no way of knowing if you have exhausted your feature space and if what is remaining is the real underlying stochasticity right we're not talking about sort of hidden variables in quantum mechanics here there is no way of knowing now however we would like to have a view of this errors not least because this feeds into trading decisions our trading decisions mean money and you know someone will come and complain about poor forecasts eventually so what do we do about it well first we need to recall why we used what we optimized in the first place we optimize the mean squared error because it's what you do for regression because that's what every single regressor does why do we optimize mean squared error well we optimize mean squared error because we are assuming something about the underlying distribution we are assuming that our underlying distribution is Gaussian we are assuming even more we're assuming that this Sigma is constant so we are assuming homoscedasticity of a data and we only fitting this mean we are fitting an estimator to the mean of this distribution and then we can do the likelihood calculation that I'm not going to go through this but if you do the maths you're trying to maximize the likelihood of the parameters that you're estimating given the data we can implement so Kerris when you do chaos loss equals MSE it really construct some computation graphs it looks like this it takes the features and feeds through the prediction model it takes a target it computes the loss and then sort of back propagates through here right except we never feed the target in explicitly but we can we can define our own loss function what I've done here I've abused caressed by passing it an identity loss function and then calculating the loss in an actual separate lambda layer right as there are actually examples as if you if you're not following all those are examples online I'll share some links to we can take it further we don't need to say that Sigma is constant we can make Sigma itself a parameter of the model and this will allow us to capture the behavior where the errors are bigger at certain times of day or for certain input features but this still leaves us with a Gaussian right keep that now there is a useful trick that was introduced sort of by Bishop of the book of the textbook Fame a while back and I it doesn't seem to be used very much but you can approximate any arbitrary distribution right as a sum of several gaussians as if we output instead of outputting a single mean in the same single Sigma Sigma we output several of those and adds them up we can approximate any distribution so you can approximate sort of distributions like this you can approximate multi value distributions where you see have sort of three components three mixture components and the color indicates the strengths the weight and the sort of switches between the components you have some switching behavior you can basically do Markov chains in continuous space okay so we can do this in chaos right so again as long as your Sigma as long as your Sigma's are non-negative your means are real valued and your weights add up to one via softmax you're good to go so we can optimize that right now in practice training these models can be a bit tricky as I do tend to get stuck a little bit you have to be careful with your learning rate annealing and the gain sort of we don't want to go into too much detail but at the end of the day so here we fitted to that previous data that I showed before and this is not a single customer this is a portfolio level right where we actually usually care about the risk at portfolio level so I've fitted a three-component I think it was three component it could be five it's just some components tend to converge to zero so you end up you have automatic selection in some sense of the number of components so here as a sort of individual Sigma's of the different components and actually you can see this is Saturday Sunday where we have much higher variability you have sort of individual means and you can compute the aggregated mean and standard deviation but your distribution is no longer constrained to being Gaussian which is very useful and it allows you to model distributions with sort of a narrow central peak and wide tails at the same time or a distribution that's not symmetric right so we can use these models for sampling and in fact tens of flow it's amazing what you find in terms of flows these days you can do sampling directly in terms of flow where I initially I sort of exported everything to numpy yeah need to as you can extract random samples as part of your model you can then compare the distributions of the real data with the sample data and most importantly you can do that all of this is has some seasonality to it so you are inevitably forced to aggregate over either certain hour a certain season but for each of those your model gives you assuming you trust your model right no one said that is a model that it removes all the if sort of cross validation requirements but you get a pretty good approximation you get this multi modality of price and this is where we're looking at this prices when you're positive and negative on the sort of last minute imbalance market and then you can sample from that now this is simply one step ahead sorry this is a sampling right from this distribution and it looks wrong because a real portfolio doesn't behave like this jittery mess now the reason it doesn't behave like this jittery mess is because it has some autocorrelation which we haven't included right so let's include that we include it was a previous time step as an extra feature if we sample one step ahead you can't quite see it on the screen but see the variance goes down a lot once we condition it on the previous point as one of the input features but then we can sample iteratively just like we do when we generate text so we can generate one step at a time and we get these smooth curves that again you can't see very well because of the resolution but maybe I can make it bigger no that doesn't help at all yeah okay I'm glad I see SS is not my strong point so yeah so you can sample realistic portfolio scenarios that are distributed in this high dimensional time city of space it has as many dimensions as there are time steps right and you can sample from that space using this one step ahead sampling and you can try to perturb it you can say oh what if my portfolio suddenly consumes five times as much as it usually does and it does tend to revert it does is the model does tell you yeah no that's not and the model just reverts back to the more expected behavior oops sorry okay so we can do this with price and volume separately and we can plot one against the other and the all looks fairly well now something that is anyone here in finance yeah so I don't know probably people in finance are very concerned about long tails and correlations and in particular sort of tail correlation saves there is a point up here where our customers suddenly consume way more than we sought and at the same time the price spikes you know this is a huge exposure to us and we haven't talked about correlations at all and I'm not going to talk much about correlation I'm just going to say that with within the same framework within tens of flobiz in Cara's we can compute things like quantiles which means we can project we can take out the seasonality we can take out the daily behavior and we can plot each point in terms of the quantile within the volume space and the quantile within the price space if we map we once we have removed the seasonality we can basically see if there is an accumulation of points in the top right corner and this is specific months some months clearly are more exposed to these tails than others so we can suddenly see those tails and these plots and shows that we need to do something we need to do some modeling to account for the higher likelihood zedzee events at the far end of the price spectrum and at the far end of the volume or consumption spectrum will correlate and of course we don't want to leave tensorflow so we just implement another model which has its own likelihood and this class of models is this is called a copula model right and then we end up with something like this and it gets increasingly complicated but actually it's not as because tenza flow allows you to construct an arbitrary graph and not just a stack of dense layers it's perfectly suited for doing stuff like this so you can so here's some volume sampling volume mixture density Network so there will be an output somewhere that has the loss for fittings that part of the network there will be an output for the likelihood loss for fitting the price part of the network there is some other network doing the copula which is taking outputs from these samples right and there will be another model so just say this is not to tell you how to do copula modeling as more about practical considerations when implementing these types of models right and the practical consideration is make use of sort of predefined layer stacks as many people have done service ResNet and the like and make use of this model sharing and sharing the computational graph between multiple models and so if you define several interconnected models you can then call each one of them and if you train one the weights become immediately available to the other and this is something that gives you a lot of power and a lot of control so what are the key takeaways from this talk and of course I wanted to interest some people in this energy space it's a very exciting space lots of things are happening with new data becoming available the markets changing in the way they operate obviously challenge your brands like octopus energy eating into British Gas and it's a big success customer base but more importantly the first point is that time series unscary forecasting time series is to logic sense a simple regression problem if you engineer your features right you usually do not need to worry about lsdm sons alike and in fact I would argue that it should always be the first step engineering date time features is something that is sort of discussed that maybe there is no standard approach you don't have a date time feature in coder in scikit-learn but you do in that github repos that are linked to deep learning deep learning frameworks yes we have several layers which makes it a deep learning model but the main power of the tensor flow and caris framework for us is that flexibility it gives us an constructing computational graphs and yes we can do CNN sometime serious but this is not the main takeaway the main takeaway is you can do layer sharing you can train different parts of the model and you've very flexible the third point is really you can embed anything so we've heard about embedding products embedding users in this case we're embedding households and their energy consumption patterns and the final point is and this is again something that I find is neglected a lot when discussing regression is the probabilistic aspect of it when you do classification you always have a soft max output with probabilities before you reduce it to a single label with regression it's more difficult to do potentially or it's done less often but in many cases like ours the prediction itself is probabilistic so when we are looking at the errors when we're looking at the mean squared error what we capture is not the fact that we haven't built a good enough model but the fact that we are trying to predict something that is inherently stochastic and making this distinction for your specific problem you may be more towards one end or towards the other end and this will sort of inform with whether this is useful to your problem but for us and especially in terms of quantifying risk it is very useful so there are some links and the slides will be online in fact actually here's a link to the slides but I'll post it on slack as there is so you can you can have a poke around this framework that allows us to build this interconnected models and connect them to scikit-learn pipelines there are some related projects that I haven't worked with personally but they seem to be thinking in similar directions to what we do of maybe okay I shouldn't take too much great we are thinking in similar directions to how tens of flow core devs think and there are some open datasets as well so you can't have our smart meter data yet but there are smart meter data sets from research projects as there are datasets for wind and solar generation by site and I think this is a great space there is not a standard time series data set like M Nestorian at right but there are many data sets out there that allow you to experiment allow you to try different things so yeah that's that with me was a constant in the friendly octopus hiding up there well and if you use this email here you can reach me and if you use this link here shameless plug I get 50 pounds and you get 50 pounds and you get hundred percent green energy and the 50 pounds that you don't get we spend it on GPUs thank you very much any questions hey you go great talk I just want to ask do you and do you how do you think about the amount of data before you kind of start thinking about regression or deep learning is that something that you take into account or is that you just have a threshold and after that point you just you can do whatever you like it's it's difficult there were several factors here why we think deep learning is a way forward so there was a great talk about profit earlier yesterday maybe profit is great if you have a single time serious profit is great it does if basically does a hierarchical basin model it works well see you don't have a scalability issue and also if you don't have this linked behavior between multiple customers right you don't benefit much from the embedding from the transfer learnings that it allows you to do whereas in our case we want to be able to have differentiable latent variables which allow us to do effectively transfer learning between customers and apply one customer seasonality implicitly to a customer who's only been with us for a month right and project ahead but also in terms of scalability deep learning predictions scale linearly in the amount of data or you scale horizontally but training itself scales sub linearly because you just you know you would think you need to go through more and more data and you do but you tend to converge a bit faster so it's actually sublinear which makes it super super attractive to us because we expect this amount of data to grow tenfold hundredfold fimo this is really great thank you and on the note of the embeddings do you retrain them periodically or you do you update them and then also have you looked at the temporal evolution and of the embedding space things yes on both accounts we have so this is what I tried to indicate here whereas the individual embed schematically admittedly way an individual embedding starts somewhere then as we retrain it moves a little bit over time yes we haven't done enough in that space because you know we have some constraints is they definitely evolved at the moment when we do reduce the full model run including all the dense layers on top we lose the original embeddings there is a very interesting paper from what was called face net where as they construct facial embeddings and they talk actually in the appendix they talk about how to update a model or implement the new model that is compatible with old embeddings is something we haven't done but it's definitely possible one final question please somebody from the back maybe I might have missed this but how do you do cross-validation in this setting okay so we do temporal cross validation we you know we train up to a certain date and then we use sort of the window months or a certain fixed size window ahead of that for cross validation during development we were working with a much smaller data set and we could do three four five fold in practice I'm not going to pretend that we do five fold cross validation everything because it does take a long time to train but on the plus side what we do we run multiple models side by side which includes trainings and monthly with cross validation on a single split and then a month later we get another split and a month later we get another split and we can effectively look back over the past six months and see which of the models has been performing better and we can on the fly simply tell the traders which model to create cool one final sorry we're out of time for questions one final round of applause for eco please [Applause]

Channel: PyData

Views: 6,936

Keywords: Python, Tutorial, Education, NumFOCUS, PyData, Opensource, download, learn, syntax, software, python 3

Id: p6mKFs6HVlg

Length: 40min 24sec (2424 seconds)

Published: Thu Jul 18 2019