How AI Upscaling Improves Weather Forecasts

this video is sponsored by you'll find out on the 18th of February 2022 the UK and Ireland were struck by storm Eunice gusts of 196 kilometers per hour were recorded on the Isle of Wight a record for England a thousand people were evacuated from the O2 Arena in London as the roof was ripped open trees were upended a church Spire was blown down in Somerset 10 severe flood warnings were issued the storm was an example of explosive cyclogenesis intensifying out of basically nothing on the 17th of February yet it was identified as a threat on the 14th and weather warnings issued on the 16th thanks to incredibly accurate meteorological forecasts only three people in the UK and one in Ireland lost their lives to the storm modern meteorology is frankly a miracle we are able to predict a vast powerful storm like Eunice before it even forms over the ocean and that saves life yet one aspect of weather prediction remains incredibly challenging precipitation meaning Rain snow hail sleet but I'm British so rain is perhaps the most challenging part of the weather to predict more than a few hours out in order to understand why we have to look at how the weather is predicted the weather is how we refer to the state of several variables in the air around us temperature pressure humidity wind speed these variables are all connected via lots of equations some of them describe how these variables are related to one another at a given location like how air pressure temperature and density are all related in the ideal gas equation While others describe how changes in these variables over an area relate to other variables for example how wind speed is related to the gradient in pressure in the surrounding area we can take the current state of the atmosphere the values of pressure and temperature and so on and using these equations predict what is going to happen at some some time in the near future it has to be the near future by the way because uh if you don't do that this approach goes very wrong very quickly doing so creates a future state of the atmosphere maybe an hour into the future that you can then apply your atmospheric equations to again to go another hour into the future and you could do that again and again and eventually predict the weather of 24 hours into the future and Beyond but to do that by hand producing a weather forecast with pencil and paper would take months historical side note someone called Lewis fry Richardson tried to do this in 1922. if you want to learn more about that you can read about it in my book firmament an introduction to a history of atmospheric science named as a waterstones book of the year and is now available as a paperback in the UK and Commonwealth and Europe I think looks all floppy link is down there in the description sorry Americans you get the paperback next year I think but the hardback is now available for you anyway back to predicting the weather to save us lots of time we represent those equations describing the atmosphere in a way that a computer can understand and do for us that means instead of describing the atmosphere over the UK for example like this smoothly varying we describe it like this we've broken it down into chunks or to use the fancy term discretized it we measure the value of atmospheric variables on this grid and then apply the equations only at locations on this grid we can then iterate into the future like I've mentioned before of course the more locations you do this for the higher your resolution the more accurate your approximation of the real atmosphere becomes in Practical terms you add as many points as your computer can tolerate without bursting into flames doing the calculation most Regional weather prediction models have points that are a few kilometers apart but why is that a problem for rainfall specifically physical processes that lead to rainfall they happen well below the resolution of the typical weather forecast computer models that we have so these processes are things like a water vapor condensing around little impurities within clouds also convective updrafts and downdrafts lead to intense rainfall all of these are happening well below the resolution of the models which is typically one kilometer for local models and maybe 10 kilometers from Global models the end user really needs to know on quite a local scale whether or not it's going to rain but also ideally how much it's going to rain and precipitation varies quite sharply over small scales and as a result these processes that lead to rainfall they are what we call parameterized and that really means that they're very crudely approximated within the models and therefore the resulting predicted rainfall can be really quite inaccurate this is Dr Andrew McRae and he's joined by Dr Lucy Harris two of the authors of this paper that was published late last year that improves the forecasting of rainfall over the UK using machine learning techniques so we use a machine learning technique called a generative adversarial Network to achieve our downscaling this may sound complicated but the idea is actually very similar to a tool that you may have used websites like image.upscaler allow you to input low resolution images and through machine learning techniques receive higher resolution images these tools are up sampling or improving the resolution of the image similarly the generative adversarial Network or gan from the paper takes in the low resolution output from a global weather prediction model over the UK information such as pressure temperature and humidity on the low resolution grid but also some extra information like a high resolution map of the height of the landscape of the UK and then does its fancy maths magic it's really and outputs a higher resolution weather forecast for the UK so it's very similar to the idea of things like image up sampling that's used in in computer Graphics one thing that's quite different than the work that we've done is that our model takes into account basically that the weather forecast can be wrong so our model we take in a single weather forecast we train it against a separate data set a radar-based data set of what actually happened and of course there's going to be differences so it's not just that one of them is high resolution than the other but it's also that the weather forecast said the rain will be here and actually it was here but that's not all there's a fundamental difference between an image upscaler and this paper and it's to do with the nature of the atmosphere the atmosphere has a very sensitive dependence to initial conditions and so if your model has taken the cloud to slightly the wrong place then even without all of this issue with the parameterizations the rainfall is still going to fall in the wrong place any uncertainty in how you initialize a forecast such as not being exactly sure what temperatures are at all your grid locations something that by the way is inevitable and the error in your forecast will grow exponentially over time that's that's chaos theory one way to tackle that is to use stochastic models where you have a slight amount of noise added to your input sort of just random noise and you see how much of an effect that has in the output so if we're slightly wrong about the initial conditions do we come to the same conclusion or are we wildly diverging away so the stochastic model tries to give a probabilistic representation of the set of possible outputs given a slight uncertainty around the initial conditions and this is how weather forecasting is done you don't just get one forecast out of your model you generate an ensemble of outcomes given slightly different inputs which can tell you the most likely forecast but also tell you how certain you can be about that forecast the tool that Lucy and Andrew and others have put together is an extension of that idea married with machine learning techniques and provides post-processing to rainfall forecasts specifically I'd say that our model does three main things really so number one it can correct biases if you produce the large number of Ensemble forecasts and took the average then that would be different to the input so that's correcting the bias number two it adds variability and number three increases the resolution so somehow our generative adversarial Network approach is doing all three of those things at once so considering how effective this tool is and the paper is clear it is effective you may well ask why don't we use more machine learning in weather and climate prediction in fact why even bother creating really complicated weather forecasting models and just let machine learning do the whole thing well there have been some very exciting papers that have come out in the past couple of years literally within the past few months there's been a couple more huge papers that have dropped one by Huawei uh with a model called pangu weather and if just before Christmas uh there was one by deepmind I've forgotten what they call their model but it's called grafcast the paper by deepmind these are purely machine learning based models and these last two papers particularly so pennyweather by Huawei and the deepmind paper they've shown that their machine learning model is basically On a par with the best existing weather models maybe better I think the rate of progress is just so vast that I would not be surprised anymore if a few years from now um the machine learning models outperform the existing models by far so now at least post processing using machine learning techniques such as described by this paper is a valuable tool in weather prediction and going forwards May provide a crucial tool in predicting extreme rainfall events and thus disaster relief as well as you know reminding you to pack a raincoat no no no no no no now I am delighted to announce that I have reached a milestone in my YouTube career I want you all to join in with this one bring up the karaoke lyrics because this video was sponsored by nordvpn we did it everyone we did it give me a high five but what is a nordvpn I hear you ask a VPN is a way of obscuring your IP address making it appear as if you're browsing the internet from another location that you can choose that has privacy benefits but it also allows you to access location-specific services for example in the UK and want to watch TV that's only available on American streaming services get not a VPN set your location to be in America want to save money up to 85 on renting a car get nordvpn set your location to be in the cheapest country for the service I've personally used nordvpn for several years now mostly to access location specific stuff and I can categorically say that several videos on this YouTube channel would not exist if I didn't have access to nordvpn's Features it's just a really genuinely useful tool but as well as being useful it's easy to use the fastest VPN out there and you can cover cover six devices on the same plan and all that costs just a few quid a month this isn't an exaggeration this isn't a bit I am genuinely delighted to be working with nordvpn on this sponsorship it's like another gemstone in my YouTuber Infinity Gauntlet I think it's a great service I use it I think you'll like it too if you would like to get yourself an essential tool for browsing online and help out the Channel please head to nordvpn.com Simon Clark to get the two-year plan and an additional month for free that's nordvpn.com Simon Clark with thanks to nordvpn for sponsoring one of my videos thank you so much for watching and thank you also to Andrew and Lucy for giving up their time to talk about their paper if you found the discussion in this video interesting and in particular stuff to do with machine learning in the future of weather prediction then on my patreon there is a bonus video with an Extended Cut of the interviews that I did where we talk about that role of machine learning and weather and climate in much more detail the name scrolling below me right now are my executive producers over on patreon.com forward slash Simon oxfiz if you would like to help me make bigger and better videos then please do consider supporting there you get access to bonus content and a monthly behind the scenes Vlog and if you support the top tier you get your name in the video credits please do pop the video a like if you enjoyed it and please do share it with people who you think may be interested in this topic if you'd like to see some more videos from me then there's some recommended viewing on the screen here and that just leads me to say thank you again for watching I'll see you in the next one

Channel: Simon Clark

Views: 19,537

Keywords: drsimonclark, dr simon clark, simonoxfphys, simonoxphys

Id: vOYnl-geOhc

Length: 12min 25sec (745 seconds)

Published: Tue Jan 31 2023