Processing Provider plugins with Hans van der Kwast

okay three two one we're gonna go all right great stuff we're starting just a little little bit early today so i want to welcome all of our qgis open day fans um to today's open day i see that there's lots of excitement around this talk um thank you abdul for your comment that you're especially looking forward to the future processing provider discussions so that's very exciting and i think you know this open day we really have a great lineup for everybody whoever wants to learn more about plugins we've got such a range of things that we can add to qgis and i think it's very powerful to learn about these things so our first talk of the day of is of course with huntsvendor quest and i'd really thank you for returning to the key just open day you're really one of our stalwarts and that's awesome um and he will be talking about the processing provider plugins um so you'll discuss sort of the multiple the multitude of tools available um specifically in the q just processing two box but through the provider plugins and then he's also going to be giving some creative tips um and just generally giving us a great overview of these different tools so it just leaves me to hand over to you hans thanks amy great to be here again [Music] i'm gonna start my screen sharing so the presentation that i'm going to give uh was also given last week at first for gnl but i made some little modifications because qgis also evolves in the time frame of one week and of course i got new uh insights during the conference so i i'm happy to share with you uh these slides which will be about qgis processing provider plugins but also provides a general introduction to the topic of plugins at our plugin festival today so i hope that helps you there's something in it for for all of us like for new users for a bit more uh advanced users and for developers so very short introduction about myself i'm a physical geographer and i work at ih delft institute for water education as a senior lecturer in gis and spatial data management of course advocate of open source software for photography and and together with kurt menke i wrote the book you just for hydrological applications and as we speak uh we're making updates and there will be a second edition coming up soon so that's a little bit of a secret i want to share with you of course with all kinds of new things that are being developed in qgis so it will be up to date and some workflows will be uh more efficient i would say oh that's so exciting yeah also looking forward to that oh brilliant so let's go to the presentation um so today will be about plugins and uh there are many different types of plugins and you can easily get lost if you are not aware about these different types of plugins so there are core plugins that come when you install plug-in install qgis that comes with qgis but there are also third-party plugins that you install through the plugins manager and these can be divided in gui plugins so graphical user interface plugins and processing plugins and there's also a category of experimental plugins but that's more the way that it's accessed through the plugins manager so when you go to the plugins manager just have to wait this uh to restart so well anyways here you see if you go to the plugins manager and you go to the installed ones in a fresh install that you see that it has already some plugins installed for you those are the core plugins just wait until it goes back to the beginning okay there it is so you can find all the plugins obviously in the plugins menu open the plugins manager and if you go to installed you find the core plugins that are already installed and an example of that is the processing plugin if that is unchecked you might have noticed that you don't have some menu items in the raster menu and you can't open the processing toolbox you can also install plugins from zip and you can also check the box here to show the experimental plugins and there are different repositories than the official one that you can add here and that's sometimes needed but where do they go many beginners get a bit confused if after installing a plugin where where does it go where can i find then the plugin some plugins the gui plugins they come with their own user interface they have their own panels and uh menus and screens like this one this is the pdoc services plugin which uh gives access to open data sets of the netherlands here we see itc and enschede where the conference was last week and you can simply use this interface to add data open data to your map canvas but sometimes it goes to a menu this is very popular probably one of the most popular plugins the quick map services plugin to get backdrops to your map canvas and after installing it goes to the web menu and after changing some settings you have even extra services added to the to the menu there are even plugins that add features to the browser panel this is an example from mergin which is the cloud service if you want to use qgis for making a field data collection apps with the input app you can synchronize qgis with through the merging plugin with the merging cloud and then to the input app but today we're going to focus mostly on this one when after installing the plugin it adds tools to the processing toolbox and you see here with the qgis logo the core processing tools so they come automatically with your installation that's also for gdal but you see here some other icons and those are from processing providers and some come with qgis when you install it and others have to be activated through a processing provider plug-in and that can be a little bit confusing and there's also a discussion about how this evolves in future versions but let's first focus what it does a processing provider plugin adds algorithms as you can see to the processing toolbox here we see it for the new pc raster tools plugin and you can use these tools also in the graphical modeler because it really integrates in this framework of the processing toolbox so here you see a little example of doing a calculation with the aquaflux function for flow accumulation with the pc raster tools plugin so we define here some inputs some rasters and then under algorithms we of course see the core tools but also the ones from the processing provider plugins and that's really powerful it means you can mix the processing provider plugins also from third-party different third-party ones with the qgis ones to create your your workflows and your graphical models so here you see in the animation the example so that's what you can create and that means you can also do batch processing it will add python classes but a processing provider plugin can also add the other things that other plugins can do like adding those menus here's an example of mixing different processing providers in the graphical model so here again you see the little icon of the pc raster tools plugin but here if you look very carefully there is the grass plugin and the combination of these different tools ends up well you start with a dm and a strala order threshold and in the end you get catchments sub catchments and channels in the vector format derived and the user can just fill in the menu and it will run smoothly because of everything being in this framework of the processing toolbox now i want to point out that there are some issues or some discussions at the moment so you might have noticed in recent installations of qgis that if you use the saga tools that you get this uh warning so something i want to point out for developers if uh and end users they sometimes misunderstand it a warning is not an error so if python gives a warning or you just give a warning or you see a warning in the log that's not an error errors are called error as you can also see in the screenshot so that that means you really need to to take action but let's first focus on this warning what happened is that qgis updated the version of saga that's being provided for a long time we used an older version 2.3 long-term release of saga but many users and developers were eager to use also the newer versions so at some point uh qgis upgraded saga and in the new installers you can only install the new version but there are some challenges there that the newest versions give issues or have differences with the previous versions and the key is in the descriptor files where some of the settings are different or like in this case where you want to do the the catchment delineation you can use the upslope tool from saga and you will get this error that the tool needs a graphical user interface and then you think yeah well i have the qgis user interface but it's actually referring to the saga user interface so that's tricky and because of these tricky things the third party tools are of course developed by a third party and it is quite a load for the curious developers to update that those tools also so the idea was to to drop these third-party processing provider plugins and have them maintained as a plug-in by their own maintainers instead of the core team being involved in that too much and there's another uh qep i have to explain that that's uh so for python you have pep the python enhancement proposals for qgis we have the qgs enhancement proposals qep you can find them in github and these are really nice discussions where you can learn a lot about the strategy and why things need to be changed in a future version so this one is even referring to version 4 of qgis and yeah this is also referring to remove non-native processing providers and we see here in the changelog of version 3.22 which just came out a few days ago that yeah that the non-native processing providers went into independent plugins for now they are in installed with qgis or you need to activate those but in the future you can expect that they will not be automatically installed in your qgis installation and you need to install the external plugins another advantage is that it gives an equal playing ground to other processing provider plugins and there are so many so in my world where i use different of these of course users find it easier to always grab the ones that are provided with qgis such as saga and grass which are really great but there's so many more things around so you can also install install the uh our processing provider plug-in use pc raster last tools or white box tools which is really cool and how do you install that so this focus a bit on the users first you need to install the third party software because it doesn't come with qgis and here you see the example for pc raster how you do that you can use the osgo for w setup under windows look for pc raster and then you install it and on linux and mac you can use conda to install both qgis and pc raster if you want to use it the second step is to install the processing provider plugin and here we see an example also for pc raster after installing the software you can install the plugin and this plugin will check if you have installed pc raster and it will give a given error if it's not installed and after installing you will find in the tools here this can work slightly different for different uh processing provider plugins because for some others so this is an optional step so it depends you need to configure the processing provider in the processing toolbox so let's have a look here in the animation at the r processing provider plugin note that there's always a home page link where you can find documentation very important if you install these plugins read the documentation see what you need to do i i now frequently get questions like hey i installed the pc raster plugin but it doesn't do anything so here you see these providers in the settings of the processing toolbox and you need to set the paths to where r in this case is installed so you need to prior to using it uh install uh software from r and then link it in the processing toolbox to uh to these executables um in order to use it so just because they went a bit quick see that again happening here so install the plugin then you'll have all the features already but you need to go to the settings of the processing toolbox and there are under providers you need to then configure where qgis can find the tools and the scripts and other things that are needed so a very important step some good practice for users the more plugins you install the slower that qgis starts and that makes sense it need to load all these things but a solution for that is to create profiles for different combinations of plugins so if you want to work with remote sensing tools and use the sap plugin and some other plugins you can simply create a profile here you can see that you go to the settings menu user profiles and create a new profile you can make as many as you want and then just install the plugins that you need in that profile and you can easily switch to different profiles by you're going to use a profile and choose the profile that you need it also stores your language settings and other global settings of qgis also a pro tip if you get strange errors then always make a new profile and that often will solve your problem because it will start with a clean installation of qgis let's now move from the user to the developer let's say you have an interesting gis tool that you want to make available you get inspired now and you want other users to use it through qgis then i think the first step and this is i'm not a developer but i went through this with the pc roster tools plugin so it's a bit with my more user-based knowledge how to to explain this but maybe there are developers around who also have great tips i think first steps to check if there's a way to uh to access the tool from qgis already is the python package already included in qgis that you use with your tool can it easily be installed with pip is the installer already available to osgo4w or can it easily be included or do you have your own installer or can you create it or do you need to run it from a conda environment which is also a great way to use different python packages and in combination with qgis and then it is strongly advised that the plugin should work like qgis platform independently so you will get more users and more appreciation if it works on windows linux and mac os then you can first test with making your scripts so in the toolbox you can add a template or use the template for processing scripts which comes with a lot of information already where you need to fill in the gaps and modify it in such a way that it executes your tool i think that's an easy step so try with an easy one and see if it runs and if it doesn't get uh give errors that it can't find your tool so here you see it again the python icon and then choose create script from the template after creating some of these tools you can then share it with a wider audience already using the qgis resource sharing plugin that's what i did so you can install the qgis resource sharing plugin and then you can add your repository if you go to settings and then you can click add repository but you need to copy from github your repository there's good documentation on how to set up these resource sharing repositories paste the github link there after giving it a name don't forget to add dot git otherwise it will not work you can add authentication if needed and then you will find there the scripts and then you can install the script collection and other users can also do that who have that github link and then they can test your tools and they'll then be added to uh the scripts section so it's not yet a plugin but it's just for other users to easily test what you made and get some feedback and you can open the scripts and see then how it works so you can also take this as an example if you want to develop your own so all these hundred scripts of pc raster the magic is done in this part where the python code is executed to do this aquaflux tool for example then if you're satisfied with how that all works you can create a plugin if you are very advanced developer you probably want to develop it from scratch and um there you can use uh uh yeah the knowledge that you have or or take an existing one from from github and modify that there's also a wizard that you can use from the plugin builder plugin very funny name for a plugin and if you install the plugin builder plugin then you can choose at some point in the wizard what kind of plugin it is it's a template to to load and you can see that you can choose here the processing provider another solution that's the one that i used is to hire a developer and can recommend experienced ones that already developed processing provider plugins there are several ones in my case was niall dawson who helped me out to publish the pc raster tools plugin and this is an easy workflow if you do it like this because you have done a lot of prototyping and it will not take too much time for the developer to turn it into a plugin and push it to the official plugins repository of qgis i had some questions from from users like hey why are you adding tools that already exist with other processing providers i can also do catchment delineation in grass or in saga i think that's not a bug but it's a feature it is i think the nice thing about open source is that you have so many choices to use in your workflow and it's because the algorithms can be different there are different theories behind it different implementations the performance can be different some can calculate very fast others are might be slower it's also about robustness we've seen these errors with saga that pop up now maybe other tools are a nice fallback option also the user friendliness or the amount of parameters for your uh tool like filling the sinks which we use a lot in hydrology can be there are already several options in saga there is in grass an option but the pcraster tools plugin has many more parameters to control how the filling is done so it's great to have all these choices what about the performance well there's uh that that could be a discussion i don't think you will use the desktop gis software to achieve the highest calculation speeds and especially if you use these processing provider tools they use wrappers and i think especially for example with pc raster tools they use the pc raster package or if you use grass then you it comes with its own whole scripting tool set which is much more efficient and even if you're proficient in python that's the preferred way if you want to make a very speedy tool but then with a processing provider plugin you can still add those tools also to qgis so i'm expecting now hopefully in the near future that there will be whole uh pc raster models added to as a plug-in to qgis which then used the python stuff of pcrust and will run fast but i think if you do that click by click way of your your workflow that maybe performance is a bit less important that doesn't mean that it needs to take hours per tool that you use that's not really what i mean but it's not that high competition about performance if we use a graphical user interface so some conclusions here i think well we know that the qgis processing provider plugins are the preferred way of adding geospatial analysis tools to qgis and it enables us to use these tools in the toolbox and the model builder which gives great opportunities it adds value to the processing toolbox and it's really great that qgis at this point is also an integration platform for many gis tools around and i think there's many more that we can integrate so if you have open source gis tools or you develop those then consider adding them to the processing toolbox as qgis processing provider plugins now if you want to find more information about that i have a youtube channel and nowadays you're used to using qr codes to show that you are not contagious or whatever so this qr code will uh lead you to the youtube channel i point out here a few uh useful ones just if you do this whole analysis workflows with saga tools then there's a workaround for using the upslope area tool uh with that error that i showed to you uh also a video about installing qgis with the old dependencies if you want your old workflows to still work uh there's a video on how to use whitebox tools enqueues how to install that processing provider plugin same for r it's about installing pc raster saga and grass tools using the the newer installers and this is a nice example of how powerful it is to to have these extra tools to derive all sub catchments from a dem using the pc raster tools plugin so that was my presentation so back to amy awesome um thanks so much hans that was absolutely fantastic um i have just one sort of question i don't know if you can answer it but it just popped into my head so why in the newer version of qgis was the decision made that you should download those plugins instead of them being sort of core plug-ins or built-in is there a specific reason that they didn't include them um does q just run foster and just sort of wanted to know the main issue is the maintenance issue so you can think of that qgs has its own base of development but also all these other tools these third-party tools have their own base of development and if you want to update something from the third party in qgis then also a lot of things on the side of qgis need to be uh updated and there is where you can get these glitches that certain tools doesn't don't work anymore or not work as expected and therefore uh from the developer's point of view it is much better to move the responsibility to uh to the third party and the the plug-in developers um so i think that's the main issue of moving them out of the core and i i think that's a good decision all right all right so totally practical decision makes sense all right and then i'm just in our comments section i see um hi wendy adrian who's saying that she's very excited for this open day in these sessions as well as some kennedy drama kayla um and i just wanted to say that i think we should start making some cue just tongue twisters because the plug-in builder plug-in for plug-in builders building plugins would be great yeah cue just plug in tongue twisters we should write a book [Laughter] all right i'm just going to give our audience a little bit of time if they have any questions for hunts um and i know hunts you'd like to finish a little bit early um so that you can go on to your introduction for your students starting the year we always call that o week here so i think that's very exciting so i'll just have about five minutes so people can sort of just ask any questions they have and then we'll let you go thanks what i'm also curious about so we've developed this pc raster tools plugin and i think it's really nice to see use cases [Music] that's something i found out after developing the plugin that's always great and it's a tool that you are looking forward to and then you can use it and i use it in my education i now also turned my new course that will start next week at iit delft using the the plugin so really using the latest stuff and work around with the saga issues that we have at the moment but i didn't realize how interesting it is for having such a thing developed to see use cases and i think there's a great occasion now in november where we have the 30-day map challenge to see maybe some use cases of using these tools or from other processing providers and post your nice map results for the 30 day map challenge i think that would be awesome maybe in next month's open day we'll have a session just on people showing off their cool use cases and their maps i think that would be great it's really cool oh that's a very good idea for a theme maybe we can ask dopey to join and give a bit of a background of the 30-day map challenge that's a very very good idea i like that idea i will take that down um that course that you mentioned is it open for anyone to do how does the course work how can people link to it yes um most of the things that i develop are open access so there's the gis open courseware platform gs opencourseware.org and i've posted it yesterday on on twitter but this course is uh is freely available it's under the advanced qgs tutorials and then it's the last tutorial i've posted it earlier but i did some extensions and updates so it's more complete now and it also includes a video for the whole process of catchment delineation it's only that part of the course that is open but there's bits and pieces scattered around on gs open courseware um if you're in a rush you can also watch the the new video that i posted yesterday on my youtube channel alrighty and i will um put some links into the description of this video once obviously the recording has been published to both um hunter's youtube channel which is absolutely amazing there are loads of fantastic videos there with great tips and tricks of course you know hans has so much experience and the best workarounds and the best like hey you being stupid just do it this way it's easier um little tips so i think that you know it's a really really great resource to check out and then i will also get the link to the course from hans and put it into our video um description so last but not least it's a little bit of an open-ended question but um for someone who is actually interested in like future processing um plug-ins and developing those but they're a bit nervous to kind of start looking at things because i know a lot of people can identify the problem they have they i don't know want to you know i'm a bird nuts so they want to make a bird map and they have a specific thing about you know immature birds versus adult birds blah blah they'd actually like to make a plug and that's all about birding but they know the problem but they don't necessarily have the skills to start doing that they don't know where to start so what is the best place to start and get advice when you want to create a processing plugin it was also something i ran into because you need to find the people who can help you with that if you're not a developer so um looking around and uh being up to date on who does what in the qgis community who are the developers check out who made already similar plugins reach out to them and i found that many people are very helpful so you can always always discuss that and i think also a good entry is the the telegram channel to find out if people maybe have ideas and maybe sometimes you find out that somebody already made a plugin for that or that there's an old plugin existing for qgis 2 that needs an update so reaching out is really the first step doing your research finding people who are interested in that and can give you advice phone a friend exactly brilliant all right so we have a question come up from um ali um thanks for the presentation it was great um is there a screenshot of qgis modeler using gdell and pc raster is it public could you share it thanks for advice or in advance um there's a a model uh repository of qgis and there's models posted there i will if i have time that's a bit the big issue at the moment i will i will add some models there that uh that use gdaw and pc raster it's a bit of an open question because you you're probably looking for an example but it's very easy to create one so if you have an idea of a workflow that needs both tools then just add those uh processing tools in your graphical modeler and if well designed it will run so there's completely no issue of mixing the g dot tools or the qgis tools or crass tools with the pc raster tools yeah so keep an eye on on updates on on the model model repository where they will be more published soon fantastic fantastic um and then there's another question from yuri or jerry sorry if i butcher your name um they'd like to know um model builder is it something like fme workbench sorry i'm not an fme user so i have to admit maybe you know emmy amy i don't know i've never used fme it's i'm very definitely a hardcore cue just fan exactly sorry no answer to that question but i mean generally model boulder is is used to bring all your elements together and often as a starting point for automatic automating processes so you know i know there are a lot of um spatial data programs and analysis options that do have something that like what we call model builder or modeler where you can you know bring in all your elements all your inputs all your layers all your tools and just make one big picture for want to fit a word and run that through so if that's what fme workbench does then yes absolutely yeah i'm sure there are lots of videos on on the graphical modeler in qgis and what it can do and i have some tutorials too about that if you want to have a look you can also export the model to python code and then modify it so there's a lot of things you can do with the graphical modeler i think it's a nice tool for for prototyping your workflows and then if you want to really make it into a tool or a plug-in then that would be a logical next step so that's also a good start if you're going on the plug-in building journey exactly fantastic all right i think that is all of our questions i want to thank our awesome viewers for logging in i know it's quite early and hans i hope you have a fantastic introduction day with your students um in the field which is even better um unfortunately in south africa it's currently raining so i won't be going outside today but i hope it's a really really awesome day and um i can't wait until the next um hugest open day and hopefully we'll be seeing you again really soon thanks amy and enjoy the rest of the the open day i'm gonna watch the recordings afterwards so looking forward to that enjoy awesome great stuff thanks so much hans cheers you

Channel: QGIS

Views: 517

Keywords:

Id: 9fWQZqPjPVk

Length: 34min 25sec (2065 seconds)

Published: Fri Oct 29 2021