Web-based groundwater information mapping with QGIS and Lizmap with Immo Blecher

all right here we go it's 5 4 3 good morning everybody and welcome to today's first session for the qgis open day of 24 september 2021 um we are very excited to introduce emma blechford leisha i hope you are pronounced your name properly uh you know and emo's going to be showing us some web based web-based ground water information mapping systems that he's developed using qgis and liz map so over to you you know yeah thank you very much tim i'm what what i'm going to do is quickly run through a couple of slides just to give some background and then i will try and run some live projects and show you what i've basically done so i'm going to quickly share my screen so that we can switch to the slideshow my screen great stuff and i'm gonna go to my slideshow there we go um just hide this great stuff i hope everybody can see that so just a little bit on the background side um you might all know that the web-based mapping of groundwater information is becoming a big trend all over the world and and and if you search the internet you will see that there are many countries that have excellent mapping websites very often those have wms links and and some of those are updated regularly and others are quite stagnant so they they don't really update any information um and and or maybe only update them now and then but if we look at um southern africa or africa in general um we don't have many of those mapping websites available and this is due to to the fact that the groundwater data is sitting in various databases and and those databases often don't talk to each other and and the data is very often difficult to get so you would have different databases for for surface water for ground water for for weather data et cetera et cetera and and and it's really difficult to to get all the data together and and obviously recent drought situations have prompted various projects from various organizations to to bring groundwater information to the web and that was done primarily to to educate the public and and also encourage participation of the public but also to initiate sound groundwater management practices and policies um by creating public private partnerships now if we look at some of the map projects um that that we have done um there were some some requirements for that and and obviously first thing was it needs to be a web-based grs map some sort of dashboard style map in in some cases um so obviously it needs to be accessible to the public and in a buzzword a very modern buzzword these days citizen science you know we try and get citizens involved um the third thing is these these um projects must be these mapping projects must be sustainable and and that is a critical part with other words we need to look into the future and see you know how long can we keep this website going and updated um long after the project has actually finished and that's that's one of the most important parts another point was that these things should be easily repeatable for other areas so if i'm doing a project in one area it should be pretty easy to to repeat the whole thing in a totally different area without too much effort then obviously the projects need to be cost effective with other words must be implemented quickly and and for that to happen the best was obviously to use open source software and and components as far as possible so what are those um com components are those open source components that that we specifically use um obviously we use the postgresql database various of those and and we we have groundwater data in uh in one database and and all the other gis layers in another database and depending on which area it is you know we might be sitting with two or three different posture connections in in qgis and obviously from the mapping side on the web and to query the data then we also use the aquabase water resources management software and i'll show a couple of links right at the end where you can have a look at where can you get these things and then obviously qgis desktop was used and then on the server side the qg server and some sort of web wetland and i often use some data management software this is optional but but it is recommended and and i use dbiva community edition or pg admin or any of the others because the db manager and q just has limited functionality and one needs to really understand the database a little bit more and and play around on the database side with some sort of tool [Music] so what are the options for for web-based maps in our case i've used liz map um which is what i'm talking about today and this map is pretty much um php javascript html built um and and and used uses those those web languages and this map also has a plugin in which you can use inside qgis i'll show that one as well and then we have qwc2 which is based on react js and open layers i have used that one before then we have qgis to wet which is a plug-in in in qgis itself also uses open layers leaflet or mapbox quite a nice tool as well and then and then you have different server versions like geoserver and map server etc etc etc there's quite a few others but those are the main open source ones um that are being used these days and all of these have their advantages and disadvantages so what and how what is needed to do this whole thing obviously you need some server pc um with apache on things web server ideally some sort of linux server you need the database in our case postgresql and that should be ideally on the same server is not really necessary but it's better to have it because it makes things a lot easier and makes it a lot faster as well and then you need the qgis server um installed in our case you need the lismed web client and then you need your qgis desktop to prepare your projects and then obviously you need your spatial layers you need your groundwater data and your roads and and and rivers and whatever you want to display um you need those so what and how is the whole thing working um some of you might know this some of you might not um but this is a good start so how to get going you need to have a running web server where you obviously install your qgis server long term release in our case the latest 3.16 um and set all your environment variables just follow the instructions on the internet it's all there you download and install lismap again follow the instructions pretty easy to set up then you would install your lisma plugin on in your qgis desktop you create your project and set your qgis server settings for all layers in the project so you must make sure that it is a valid future server project and there is a little test function on there i'll show that as well for those that don't know and then you run the list map plugin and configure the layers and the map so on the on the server side you would then obviously copy this whole project file in the list map configuration file that you have on your desktop you copy that to the web server i use ftp for that that's the easiest in the lisma admin interface you create the so-called repositories basically those are the folders where your project files are located you can also set a few settings i'm going to show that as well then you create the content for the landing page that's basically the one that shows the different projects on some sort of a web page when you go to any domain which you have set up and it shows the projects on on the first page and then you can optionally also create users and groups with different permissions for access to the maps and the layers so where did we get all the data obviously the point data is our borehole springs wells etc etc and those came from most of those came from the national groundwater archive which is sitting at the department of water and sanitation um but we also got quite a lot of data from the city of cape town and then obviously project data so a lot of consultants in the areas they have done various projects and they have created spreadsheets and all kinds of other access databases etc etc all those were collected and they were imported into the same possible database chemistry data for groundwater quality etc those came from the water management system also from the department of water and sanitation and also from the city of cape town and then we have rivers dams measuring points so those are basically points where samples are regularly taken or measurements done in rivers or dams and those came from dws also the hydrology section and then we also use rainfall data because people are very often interested in rainfall what's happening with our rainfall so those came from hydrology and then also various internet downloads then the normal layer data is hydrogeology rivers dams etc a lot of those came from also from dws and then also osm osm downloads and then we have different base maps so some of those are built in into the list map functionality but also the ngi which is um the [Music] also a government institution which has a lot of layer data available also for download they've got a very nice portal as well so what does it look like i'm gonna switch now between basically this slide is is just giving me some sort of hint what i want to show um so i'm gonna quickly run through the the qhs desktop and then also show what the groundwater data looks like then i'm going to quickly show the lisma plugin and then some projects the list map admin interface and and also the the actual maps so let me quickly switch to my is the hd right um i'm actually using uh two different versions of um of qgis on my pc i prefer always to for my projects work in the latest versions i've got the 3.20 um and as you can see i'm working on on ubuntu linux but i also have installed the long term release with the with the flat pack so that i can do the projects for lismet because lisma it's very finicky it prefers to to work with the long-term release versions of qgis server so this is what i've also installed on the on the server side and on the desktop side i'm also using the long term release the latest long term release version so as you can see i've got one of the projects open here on the left hand side are all my my different layers that i've got available and it actually doesn't matter what is switched on and off here in this project because you will see that in the lism plugin you actually then configure what you want to see on the actual online map so what is important on this side is also that you can add more base maps because there are a few built-in ones but as i said you know you can also use all kinds of other wms maps that you can add to your to your project and you can use them as as base maps and just very briefly to the qgis project properties obviously on the qgis server side so in this tab here make sure that you fill in a couple of details there on the top but what is important is that here at the bottom the waf capabilities um those need to be switched on specifically for those layers that you actually want to use for presenting on your map um your online maps so these these are very often if you forget to switch these on sometimes you won't be able to to to see anything or to do specifically the charts in the list map functionalities right at the bottom here you will have a a test configuration you can launch that and it will tell you that the project is valid the project is not valid you might not even be able to publish it properly on the um on on on this map and on the web on the webbing web map site so what is the list map plugin doing you can switch that on or click click on the little um plugin button you will see there's on the left hand side there's various map options here you can set your extend switch on a couple of things that you want to see right in the beginning and but what is also important is you you can set here access and restrict access to certain groups of people and we're going to come to that when we look into the list map admin interface so here you can define groups that can actually see the maps then you have all the layers here you can obviously decide which ones do you want to switch on and off right in the beginning so if it's toggled then it's switched on if it's not toggled then it's not switched on so that you can set up for each and every layer that you wanna set you can do here then you have the base layers and you will see what those are and how they display um i have for example open street maps and and the simon toner but also switched on the google street satellite the hybrids and the terrain obviously you need your api key for that and here at the bottom you set which is the active base layer on startup then there's various functions here locate by layer this is just for searching data on the web and you can tell it which tables must be available as attribute tables which you can download you can also enable layer editing which i haven't done you can set tooltips for certain layers form filters you can filter by user there's a few new functionalities down here then there's a database um part which is a which is a charting little charting tool um which works quite well if you know how it works it's a little bit of fiddling around but if you know what you're doing you can get that going and i'm going to show a few examples but it also has got a time manager so if you have got temporal data you can use that as well and it has also got an atlas feature where you can show different different maps based on the atlas that you set up in qgis so that is basically the list map tool the list map plugin so if we can go to the first prod first project now first of all you can you will see that um this is our the so-called landing page so i'm giving a short introduction of what we're using how we do it and obviously a disclaimer in terms of use and then you have available projects available here at the bottom now in my case i've got two here and i'm gonna show you the first project now before i switch over and and switch log in as a different user so what you will see here is and and these are basic these two maps are available to the public already they have been done for two projects quite a while ago in fact this one was for the city of cape town and in environs that was done two years ago almost two years ago initially i did this project on qwc2 and what that looked like i'm gonna quickly show you this is what what it looked like i just had a few screenshots on that um it's basically another web client um a little bit more tricky to set up i found that we had to do quite a bit of customization on the on the different source codes of the different modules um and i found that the qwc 2 was quite difficult to update because because we had so many little changes to the source code we we found it difficult to to maintain forward so there was a lot of work involved if you want to upgrade to the latest versions but it was quite a nice um web client it basically has got layers and and legend on the right hand side you will find the same layers i used in this map it pops up info here feature infos another example was this one you could also set um different base maps and and also have different layer info and polygon layers search functions etc etc so that was a nice and i believe is still quite an active developed client and one can have a look at that in future again um so what happens here is on the left hand side is basically like in qgis is your [Music] your layers you can switch them on and off um as you can see these are my different uh sites here for example we've got um river stamps pans canals monitoring stations etc etc these just gives you an overview and the same as with boreholes you can click on any of those and it will bring up a little pop-up and tell you where it is what the longitude latitudes are what site type it is etc if you switch on we had quite a few different ones here also monitoring networks now one of the monitoring networks is the dws hitstra maestro water level monitoring network that is quite updated the department of water and sanitation measures a lot of these boreholes on a regular basis for water levels and some of them also for chemistry and that data is readily available for download from the website dws website the national groundwater archive but you for this project are also used because it was a citizen science project and we wanted to to show the the public um what is available we have for example river stage height and flow monitoring stations you can when you by using this little button you can always switch on the um the legends as with for example for boreholes different types of boreholes well points but we for example have unverified real-time stage heights these are quite interesting because you can click on here as well it will tell you where it is so this one is in the in the berg river and then we provide on sort of links to to online charts so clicking on any of these links will bring up a little chart as you can see today is the 24th so it will show you a chart of the berg river um for the last 14 days now a lot of these um you can also see the the flow chart for that um so this is in cubic meters a second and you can see that the flow decreases slowly so there was obviously a rainfall event here and then your flow decreases now these are just uh some live maps but as as it says it's unverified data so what happens is that once the data is verified it becomes actually downloadable from the from the internet from the dws website and you can plot them as as charts now the same applies to dams um same thing and this was a critical one when we had the dams also always almost reaching [Music] reaching a percent levels um in 2018 so there's still about a slow of them if we click on that now you will also see i'm also getting links um same thing here um this is the measuring point in the tibetan sleeve of dam and the interesting one is obviously the capacity so there's 100 capacity line and anything above that is obviously overflowing and anything below that uh so we are sitting here today at around 99.2 yeah a roundabout right so this is a little bit outside of groundwater um if we go to the home page there are the two projects so if you want to memorize this one it's www.groundwaterinfo.africa so if i connect i'm going to log in with admin now you will see that there's a lot more projects only available here even some that we use as admin here for testing and i'm going to show you a little bit now the first project that you've seen was the one for the water research commission which is already a couple of years old but i try and keep it updated so the nice thing about this is as soon as i add any data to the groundwater database it immediately will show on the map so if i add another borehole or two or ten which lies in that particular area they will immediately show up on the map now if we zoom in quite a lot quite a lot then we get to another project which we have done for a mining area in the langeban area so this information is obviously not available to the public and you need to have a login to to get to that particular map um but as you can see um in the legend we've got the mine here langaban is there and and there are various monitoring boreholes that that are used to check what the water levels and the quality is doing so here we use the database charting facility quite a lot and we are still working on it i'm going to switch off a few here but for example if i click on this one you will see that it gives me again some basic information there on where the site is what it is what kind of purpose the site is used for and then we've got various charts here at the bottom this one is for the quality obviously our electrical conductivity that was monitored from about 2017 until today so about four years of monitoring and you can show different limits that you want to achieve or not want to achieve and the same applies here to the water level so in this case we are measuring the the quality and and we are measuring the water levels in the area and that you can do for any of these uh ball holes here as well so if we go into the mine area itself you will see same thing happens here water level declining why is that declining because we want to get the where the actual pit is this is where it is we want to get it dry so how can we present a little bit more here in terms of background this is osm background if you look into the layers here right at the bottom there are the base layers so if you click on those you will see there are the ones that come with liz map which i switched on in the lis map plugin but i added those here as well so you could for example also add the ngi imagery to that and there is the actual mind that that we are monitoring with those boreholes um now we've recently also started around about in april another project which i'm doing um together with uh frank frank zuckerlitz um and frank is is uh by the way he twisted my arm to actually do this presentation today so thanks frank um if you're listening um but this is the latest project that we're doing for the wwf and which takes a little bit of a different approach it has got a sort of a dashboard built in now there was some research done what do people want to see on a map what do they want to see on a dashboard and this one specifically is for for groundwater in the table mountain water source area it pops up a little info introduction here in the beginning if you click on ok you will see that these are the three different [Music] water source areas in in the cape town area so we've got the table mountain in the table mountain and cat peninsula and the red is the kite peninsula and flats groundwater and the all these three have different properties i'm gonna switch them off now so that you can see a little bit more of the boreholes um so the project is about also about monitoring so we want to monitor different areas and and show the public what is actually happening happening in these areas what's happening with rainfall what is happening with our own groundwater levels because more and more people start using groundwater and obviously if they live in these particular areas they want to know what is going on around them so we've taken the database um and and uh frank has done quite a bit of javascript on this and to to display um to display some information on this so what we've done is if you look at the complete area that we are showing here the total area with all the boreholes that you're seeing here those are the ones that we are monitoring you would see that we have average water levels that we display here um in these little blocks so you immediately get an idea that our latest is now six meters last five years the average was 6.1 so we are pretty close last year this time the water level was a lot higher and in the last three months the average was about 3.8 it was also a bit higher than the latest um now this obviously uses the latest data so if we if we add more data to it you know these these figures will immediately change so if i add the latest downloaded data for these monitoring boards here this this value will change immediately and then there's the water level chart which is the daily average of everything here that you see on the map there is some data missing there which dws knows that it's not there but they don't know where it is interestingly enough and there's also a rainfall chart here which is actually quite goes back until from the 1950s um to very recent now what we've done now is we have identified certain monitoring areas and and we are adding as we go along in the project there will be three more monitoring areas added at this stage we've got three of them so i'm gonna go and choose one of these monitoring areas new lens everybody knows where that is so it will zoom in there and you will see these figures down here at the bottom immediately change so now it's only taking those four holes here which are sitting in this monitoring area to calculate these values so now we are looking at the newlands monitoring area also you can see the water level chart changes so that it only shows for the monitoring balls in that area and the same applies to the rainfall only filtered according to the monitoring these areas now you might ask i don't see the rainfall okay let's switch it on if we look at the our other geosites here our monitoring sites obviously is the important one but if you switch on the rainfall stations you will see that these are represented with with pie charts and these pies are also done in qgis using the chart to their little chart facilities in your symbologies so what we've done is you can see immediately the size is obviously the the total rainfall so that if i click on on any of these will pop up my information of my rainfall station this is obviously my table mountain location in new orleans and it will give me for january february that the the averages per month which are displayed in the pies it will also give me the annual total and it will give me a chart at the bottom as a as a bar chart um over the rainfall now you can zoom in into these areas etc etc so there's quite a lot of things that you can do in these charts but for producing this rainfall chart and these water levels here it's only using the sites which are in this little monitoring area so if we switch the monitoring area to something else which is for example the philippe area you will see these figures will change now now there we go now it takes into account only those philippi boreholes and these are mostly the ones that the dws monitors quite updated data i think until august almost and obviously only these two stations there rainfall stations so this is a different approach again so you can see that um with these uh three three projects that i've quickly shown you you can you can do all kinds of uh wonderful things and one of the nice things um is the data functionality here so i could go on to any of these monitoring boreholes here that we've just seen and i'm going to click on that one and it will give me my daily water levels here and i can actually download them and and yeah and the same you can obviously do with we're gonna close this sorry i'm gonna go on to the um data and i'm gonna you can use any of those you can go to the rainfall stations here as well click on any of those and you will get your rainfall data which you can download so that is our data facility the charting facility what is also nice you can draw on the map if you want to present something to somebody or show you can draw little lines polygons etc but you can also import geojson gpx or kml files to display additional stuff on there you can print the chart and if you actually use the geolocation tool which is also quite nice now if you use that you can actually zoom in quite a lot and switch on the geo location and you can walk basically to a borehole by switching that functionality on the information panel is the one that gives you all the information what the project is about and what is um also quite important here in this case is um any of the layers um on any of the layers you can if you click on the actual layer itself it will give you a description on what this layer actually depicts what it shows so what is a borehole etc etc but the little i or this documentation button will also then jump to additional information so the public can immediately read what is a borehole it takes it to the department of water sanitation groundwater dictionary so you can read up on additional things like what is an aquifer etc and that applies to all the layers here as well now one of the things that people are always interested in is obviously our groundwater occurrence where do i find groundwater and what are my borehole prospects let's switch that on if you click on the legend there you will see that there is certain success rates of drilling a successful borehole and also how strong it might be as soon as you switch under layer obviously the background disappears a little bit but you can switch it on and you could for example choose a different capacity by which you can still see the background map and have the other boho prospects overlaying onto it so there's lots of possibilities with this i really can recommend using lismap for something like this because it's quite user friendly quite easy to set up and yeah it's quite an effective tool to use to present not only groundwater data but any data on on the on the web browser so very briefly where do i get all this um qgis everybody should know by now where to get it this map is on the listmap.com website if you want to use um the the groundwater management software just go to the aquabase.pleasure.co.seta website where you can find additional information on on the software and you can also download the software there it's also open source the only thing that um here you will find on the home page you can go to the downloads and you can download it and then obviously for postgre you can go to that website and download the database echobase can use different databases basically to store not only groundwater specific data but also rainfall dam levels river levels etc etc so in a nutshell this was my presentation thank you so much you know you might have noticed we were joined during your call by another guest so maybe i can introduce i would say a large part of the brains behind this map is that fair to say it in and hi everyone and i i thought we could maybe just have um some comments from idiom having watched your presentation maybe he's got some suggestions or tips or or thoughts on what you presented yeah so i'm part of freely's like the company behind this map so i'm not the author of this map it's like we are in the team and thanks a lot for your presentation because it was quite nice and i could notice that you were using like our release come to date on a production server so you like adventure yes hopefully we are going to release this new version soon uh we just merged the last feature this week so we are planning to release this map 3.5 uh yeah end of september beginning of october and i noticed that you also did like yeah like some very nice presentation about database i guess you you went uh with a lot of features and research to make the database working properly yes um we have used it extensively and and and as you know you know every time something works or doesn't work you know i quickly jump on the net and and send out help please help yes exactly we have yeah it would be nice to be able to use that absolutely uh because it's the same engine behind yes so it would make it like easier to try first in qgis desktop your plots with that absolutely config and then having like it's part of the roadmap too like it's a wish it's not easy to get there but it's it would be nice yes but i can promise you i will i will test it more and i will probably at some stage also maybe make some feature requests you know if i find something that might be nice to to add as well sure thank you thank you it's a it's a very nice very nice feature very nice plug-in and a nice tool to use to produce maps on the internet very nice sure you should you shouldn't hesitate to create tickets on github and we try to answer or to build the community with users and translators and it's all open source everything is on github so like there's translation bugs feature requests and it's not easy to understand sometimes also like a list map you need like qg server you need like the php application you might need as well qg server plugins uh yes you add plugins on qg server you will enable more features in this map to have like more extensions where you when you download data from this map or now you can have list map qg server plugin as well in qg server you can then like evaluate qgis expressions so if in your forms you have some constraints or some default values or some geometry checks uh the list map plugin is used on qg server as well to evaluate the qgis expression and return the value so it's not easy sometimes you understand but we are trying our best to to make it clear yeah now we are getting there yes i think you are muted sorry about that so we have one question from the chatroom from ingestion i don't know how to pronounce your name i'm sorry if i said it wrong uh he's from southern spain he wants to know um in order he says in order to prepare a similar river basin project the main concern is about keeping data which shouldn't be available to the public um is it safe enough in list maps so i guess he's basically asking about the permissions framework and if you can hide or make certain parts of the data accessible only to authenticated users yeah so there's different levels about authentication like you can have like projects which are private so you like some users might just won't be able to see them you can also have permissions at the level at the layer level so you might have in a single project having some layers which are visible or not according to if you are in the admin group or if you are just an anonymous user or a logged in user and there is also like per feature level permission so if you have like in your attribute table you can set like some groups or some users which are enabled like where you can like say okay this user can see this feature and for the next version of list map 3.5 you can now have like a filtering like per uh polygon so if you have like a municipalities layer with municipality abc you can have like a point layer uh like uh shops or like uh and then like you can set like okay only this group of users can see the features in uh the layer uh a or like i mean the municipality a or b so you can do some special filtering uh so it's all the filtering is done in qgis so if your qg server like is not available like every request should go through this map and list map will just make like a kind of proxy to check like user permissions and only this map is allowed to request data to qg server so it's pretty safe to use yellow if your qg server is not available like online straight like if all request goes through this map it's all right so you've got a kind of a reverse proxy yeah between this map and curious server um one question from me is just about the theming is there in like a place that you can download and find different themes for this map or like um that's a good question we know that we must spend some time on theming and just updating to latest uh like trends in the documentation there is a link to how you can customize your team some of the settings are already available like straight from the administration interface to change logo and just the colors but then if you want to go further you can like there is in the documentation there is a link where you can download the full theme uh used on this map you can customize the css and then if you drop it on your server like not in the list map source code but in your project folder this map will first try to check if there is a custom theme and then it will override the default one we don't have a library of theme available we have a library of a library of javascript script i think you may have used one or some in your presentation uh but for the theme yeah we don't have it it's just spread online yeah that's um basically what it is you know at this stage in the admin interface you can't play around with colors etc so if you if you don't like the the standard grayish sort of i changed also some colors but basics can be done um quite easily but as etienne said you know you can also go a lot further with that if you if you download different themes so we're coming to the end of our session maybe i could close out the session by asking you each to give a one minute top tip for somebody new to liz map like what's the like your best tip for a new user let me start with you you know yeah this map is is is quite a versatile um web publishing tool um highly recommend it um because it it it works quite well with with qgis obviously um if if you're comfortable with qgis you will understand list map and and the nice thing is what you prepare in your qgis project is exactly what you will see on the map so the colors are the same the sizes are the same everything is exactly the same so i would say this is um this is an ideal combination of of qgis server qgis desktop and and this map to publish maps on the on the internet thank you and yeah one tip is uh it's quite big but there is a docker compose project to try this map so because you need a server with the php application running on a server but if you want to try on your laptop then there's a a docker project that you just run and you have a list map running on your computer to first give it a try and then like if you like it then you insert it somewhere but first if you want to try uh think about docker there is one one link where everything is set up inside is the docker configuration good for production use as well or yeah yeah we are using it in production because we are also providing hosting solutions about this map so we we asked many customers and we are using it in production using doco okay thank you and thank you both of you for your hard work and contributions and uh for sharing all the work you've been doing you know it's really great to see um such a comprehensive and complex application being developed and on top of this map so it's really nice use case and we look forward to hearing about your updates and some at some stage in the future thank all right thank you so much i'm getting in the stream there okay thank you jim thanks a lot bye-bye

Channel: QGIS

Views: 1,441

Keywords:

Id: hnMncLhqmrk

Length: 54min 45sec (3285 seconds)

Published: Fri Sep 24 2021