Download ERA5 reanalysis from the climate data store

[Music] hello so now today we're going to actually start to look at some of this climate data in the re-analysis essentially we're going to be looking at how to access the era 5 reanalysis and through the web interface so what we're going to do is we're going to open a browser first of all so i'm using the chrome browser on my mac laptop so you can open chrome or firefox any browser that you want to use for this exercise and i want you to basically go to the copernicus website which i will put the address in a banner at the bottom here but it's cds.climate.copennicus.eu okay so we've got this additional slash home here but you see that just comes up automatically so you should see a page like this which you have the banners across the top of the european union who are funding this development there's copernicus eastmwf and this is under the auspices of their climate change services now if you notice before i go into the details at the top right of this page you'll see that we have a button that says login or register right here so if you click on that before we start you'll see now they've got a login page and i've already saved my login email address and my password but for you that those of you that haven't used this service before you'll need to click on this tab here now create new account and you'll be faced with a page here with information that you have to enter so your email address you want to use to access the system your name surname country and sector okay organization so there's not much information they need to use here you might want to read the terms and conditions about how the data is permitted to be used so after you've done this you only need to do this once obviously to set up a an account uh you won't need to do it again and to have free access to this data so once you've done that and confirmed you should get an email that arrives in your inbox as usual for this kind of like registration which you need to click to confirm your email address and then you'll be able to in future login so here i'm just going to click on log in so you might need to wait one or two minutes to get the email okay so pause the video until you have that set up so once you're logged in instead of having the login button here you should be successfully logged in you'll see your name displayed on the top right corner here so once you've done that we're going to start to dig into some of the data sets okay so i'm only going to scrape the surface so what we're going to do is we're going to dive straight into the database which on this menu across the top here we'll see that we actually have the third from the left we have data sets so i'm going to click on that and now we get this page which has basically a search button okay and then on the left we have a a menu a list of drop down menus that we're going to come to in a moment and we basically have a list of all the data sets we can scroll down and you'll see there's a large number of very diverse data sets here okay so we're not going to be able to go through all of this this list is growing all the time as new data sets are added for various different climate sectors and services some of which you might find very interesting so i invite you to browse through these we want to go for the basic era 5 re-analysis that we discussed in the last lecture so if i click on the left hand menu here that says product type okay this will help us reduce this menu of options here to something that's a little bit shorter so if we drop this down we'll see that there's a number of different choices we can make there's climate indices climate projections external services in situ observations reanalysis and satellite observations and seasonal forecasts that we're going to talk about in part two of this course so for the moment what i want you to do is click on the analysis where we'll see that of these data sets 21 are actually classified as urea analysis which means they're either directly the re-analysis or they are downstream products that have been derived using re-analysis data so if we click on that we will find now that we still have a rather long list but it's shorter than before so for example this is an example of a downstream product thermal comfort indices derived from era 5 reanalysis but what we want to do is use some of the direct era 5 data so you'll see that some of these data sets start with era 5 and the difference is they're all from the same underlying data set but some of them have been for example aggregated over periods of time such as we have monthly average data here and you'll notice that we have basically two types of should we say locations in the atmosphere which the data is available okay so the first type is data that's available on pressure levels okay so we have slices through the atmosphere at constant pressure the data has been interpolated from the original model levels of the actual simulation grid onto should we say isobaric lines of constant pressure okay so on pressure levels or we can have basically data that's not a function of height and therefore is data that's available on single level so here we have monthly average data on single levels okay so single levels is usually referring to data at the surface such as the temperature at two meters height okay but it can also be data that's available at the top of the atmosphere so for example top of the atmosphere radiation fluxes infrared fluxes solar fluxes and so on okay if we scroll down we'll see lower down in that list we also have some error five products that are available at much higher tempo resolution so hourly data so if you're interested in daily averages of data and so on you're going to have to use hourly data because there are only two levels of temporal aggregation there's the hourly uh available data or data that's been aggregated up to monthly time scales there's not directly available a daily average data okay so if you want daily data you need to take the hourly output and actually aggregate it yourself offline and we're going to be showing you how to do that in the next lecture so just as an example here i'm going to click on era 5 hourly data on single levels from 1979 until the present okay so if we click on that you should be on a page that looks like this where it says era 5 hourly data on single levels from 1979 until the present and underneath that you'll see that there are a number of menus overview download data quality assessment documentation and view now if you're unfamiliar with all the intricacies of the era 5 reanalysis i do think it's worth time having a look through some of this documentation because it will really enable you to appreciate some of the limitations of the data some of the strengths of the data how it's created and so on so there's a lot more detail available in some of this documentation i was able to actually cover in my um last lecture so you see we've got an overview of the data here and then there's some data tables at the bottom and these are particularly useful so you have got a list here of the variables are available within the data set and in this table as well there are the units so if you're ever confused about what the units are of the the variable that you're looking at although we'll see they're actually stored in the file themselves and again in the next lecture we'll see how to see that information in the metadata that's record and then there's a description of the parameter so it's it's it's very well documented and you'll see here there's even more detailed documentation more generally about era 5 documentation online so we're going to move on quickly though i'll leave you as an exercise to look through some of that documentation and we'll see here that there's a tab download data so we'll go straight to that so i'm on download data under error 5 hourly data and single levels from 1979 until the present so make sure that your screen is showing the same information and that you haven't accidentally clicked on the monthly data instead of the hourly data what you'll see then if you do that is we have a number of these boxes which we're going to go through quickly in turn so on the top we've got re-analysis or ensemble members or ensemble mean or ensemble spread so if you recall in the last lecture we talked about the fact that now era 5 is not just a single analysis but it tries to account for the uncertainty of the analysis by actually having several different should we say analyses possible say manifestations of what the atmosphere is doing to have an assessment of the analysis uncertainty for the moment though we're going to restrict ourselves at the beginning of just the basic unperturbed analysis so the re-analysis product so for most people this will be sufficient uh compared to for example forecast errors and so on that these analysis differences are actually quite small but you can if you're interested you can have a look at these as well to have an idea of what the uncertainty in the analysis is with your particular variable what we're going to do next then is have a look at which variables to download now you have a lot of drop-down bars here that try to group the parameters in categories of retrieval which are fairly self-explanatory so we have precipitation and rain categories and so you can see that you have for example the total precipitation or you can have it divided into the convection uh parameterization schemes rain rate and then the large scale rain rate and so on and uh if we look at clouds you'll see that these are all variables that are related to the clouds in the column of the re-analysis we've got the cloud-based height the the low cloud cover total cloud cover and so on okay for your convenience uh obviously from experience with how the system is used there are a number of these variables which basically uh are very commonly used and so these are um say put into a separate table at the top here in order to make access to these very very fast so we have the 10 meter wind on the u component the 10 meter v component the two meter dew point temperature uh the two meter temperature and so on i'll let you read through those so these are variables which are very commonly downloaded and so you've got this kind of quick access here okay so in fact for this first exercise we're going to pick the re-analysis and then with this check box we're going to click on the two meter temperature okay just to start us off if you want you can actually select more variables but obviously it increases the files file size afterwards now one thing i should say actually when we get down to these last boxes here so this is now we're starting to look at the dates is there is a limit the amount of data you can get in one request using the web interface so what that means is that often you might find you need to break down your requirements into several smaller jobs now that means if you're using the web interface a lot of clicking and so on and so what we'll find later is it's actually most convenient to use the api okay so you can actually retrieve the data directly to your laptop using a little script in python so we're going to come to that in a future lesson so the web site is good for investigating the data and maybe getting one or two quick downloads of data to enable you to try an id out but if you really want to get a lot of data you don't want to be using the web interface eventually you want to be writing an api based script which we're going to cover in a future lesson so what have we got so far we've got a reanalysis two meter temperature and then lower down here we have to actually specify the dates and again this is fairly self-explanatory we have a number of years here the re-analysis now goes back to 1979 so it covers the same period as era interim okay that was the predecessor of era five i believe the plan is actually in the future to start extending this back further into the past now one of the problems of that of course is when you start to go back before the 1980s there's very little satellite information available so from the 1960s and earlier than that you really only have the conventional observations if you remember from my first lecture of radio suns and in-situ uh ground-based data and so that means that the it gets a little bit more difficult analyzing trends when the data information is ingested into the system changes over time okay so mostly uh most analysis systems like euro interim and this actually start from 1979. so for this we can see it goes up to 2020 it's not up to real time so if we click on 2020 we'll find that it's just a little tiny bit behind real time but it's not too bad now it keeps fairly up to date so i'm clicking on october now we're currently on the 23rd and we can see we've got available data up until the 18th of this month so it's only about four or five days behind real time now which is uh is actually very very good indeed in the past there will be a much larger lag this also shows another aspect of this system which is quite nice in that if you notice when i make choices if there's an option in a table it's incompatible with another one so now i've got 2019 and 2020 i've got all of the months which are dark and clickable when i only have 2020 if you notice november and december have disappeared because obviously they're in the future the data is not available yet so the system prevents you from making choices which are not compatible with the data that's online but obviously if we try and download november and we click 29 in 2020 now 2020 is not available for november and it disappears on the other hand if i click may uh when we actually have both months so now if i choose these two months 2019 2020 may and november of course the retrieval will only cover three of those four possible months because november is only valid for 2019 and not for 2020 so you don't always get what you expect from the combination of all the tables as you can see but it's very very helpful nevertheless that you have these options that are writed out so we'll just pick up one month of data just for this very first initial test so we take 2020 i'm picking may now if you notice days well we could actually take all the days and you notice that instead of clicking all the days like this we have uh these buttons here like select all which enables to very quickly select all of the days here and if you recall the data is hourly so we have another table here that enables us to actually select specific hours now what i'm going to do is i'm going to again select all of the hours we've got all of the hours for all of the days of may 2020 for the two meter temperature now here we have the geographical area now in the web interface this has been recently added in the past it was only possible to get global data through this and so if you're only interested in having data for a sub-region you can click on this and actually cut down the error which of course will drastically reduce your uh file sizes uh depending on what the size of the sub-region is obviously so you can use that as well for the moment i'm just going to try and click this for the whole region and last but not least we have the format of the output so we have grip and net cdf so this is the format the output now you might not all be familiar with this kind of data outputs basically both of these are self-describing files and i'm going to be talking about that kind of file in the next lecture when we start to analyze the data okay grib was traditionally used in uh operational weather forecasting and net cdf was a file uh should we say self-describing file type that's more generally used for climate so we're going to click on netcdf and at the bottom here then we have terms of use with a license agreement now if you're using it for the first time this won't be ticked okay and you will have to click to read the terms of how the data can be used which i think has conditions such as you're not allowed to sell the data and and make profits from the data uh without the explicit permission i presume of ecmwf so you can read and view the terms and you have to click that you only need to do that once and then they store it under your profile and so in the future you'll find that this it will be clicked for this particular data set so if you use the seasonal forecast later you'll need to also agree the license for that and now i'm going to submit the form so here we go i submit the form and then you should find that it goes to basically this screen here where we have uh this is my retrieval name the submission date okay and then the duration and it says in progress so the system's going to the tapes now and it's actually trying to uh shall we say pick this up you can now if you want go back to the data sets page and actually choose a different um should we say data set to download and that will still be running in the background okay so it doesn't kill it off if you move away and in fact if you want to go back to that page and see how your retrieval is progressing you'll notice two tabs to the right of data sets you have your requests you can click on this again and we go back to that page and you can see it's still in progress now we have to wait a little while uh it depends there's a queue you need to rise to the queue for the to be processed and to actually retrieve the data into a location where it can be downloaded so you can see i've got an example here that's still stored in the cache that was actually requested a few days ago and when it's ready you get a green download button just like this one here so i'm going to leave it there for the moment i want you to attempt to launch your own retrieval and then when we come back in the next lecture we'll actually start to look how to download and make a basic analysis and start to manipulate these files thank you very much and i'll see you in the next lesson [Music] [Laughter] [Music] you

Channel: Climate Unboxed

Views: 19,719

Keywords: era5, ecmwf, download, netcdf, climate, weather, CDS, climate data store, Copernicus, web interface, downloading

Id: AXG97K6NYD8

Length: 20min 41sec (1241 seconds)

Published: Wed Mar 24 2021