A Tour of QGIS Processing Toolbox Algorithms

right hello everyone greetings from india my name is ujwal gandhi and thank you for coming to my talk today i'm going to talk about processing toolbox but before i start i would like to thank the organizers for making this conference really accessible and allowing this opportunity to come and talk to you though i'm located thousands of miles away from north america in this talk i'm going to give you a tour of some of my favorite and lesser known toolbox algorithms a brief intro about me many of you may have seen my tutorials at qgis tutorials.com i've been writing and maintaining those for over a decade now i also run a company called spatial thoughts which is completely focused on training we are one of the qgi certifying organizations and offer online training for qgis python and google attention i'm also a visiting researcher at university of jonas work where i teach urban planning in gis using qgis so the structure of the talk today it's going to be hands-on i'm just going to do live demos and cover a whole bunch of tools so let's just dive in this is qgis 3.14 you can open up the toolbox from the processing menu the toolbox contains a variety of tools they are organized by providers the tools that start with queue they are the core tools this ship with your qgis installation and they are written and maintained by the qgiswpt you also have other providers such as grass and saga these are other gis systems but you have access to those tools via the toolbox allowance you can also install plugins there is a new plugin called whitebox tools you can install such plugins and it adds more tools to your passing toolbox you can run a little script here that tells you how many tools you have and in my current qgis installation i have over a thousand tools that means you have a wide range of tools that are available to you to do a variety of tube processing tasks and this talk i'm going to give you a glimpse of some of the tools and that gives you a flavor for what kind of things you can do with the toolbox we'll start with this tool called create points layer from table and browse to an excel file and this tool allows you to bring non-spatial data such as file data in this excel file as a spatial layer qgis and the structure of this file is i have a latitude and longitude column which i can configure here and can import this file as a points layer if you have to work with some business data or even some weather data that is in such formats as a csv or excel file this is the easiest way to bring them in in a single step from excel file to your points directly in qgs great so we have a points layer now let's bring in the u.s states layer next maybe i just want to work with the weather station in this one state you can do this in multiple ways you can create a new layer from the selected polygon select all the weather stations that fall within that and save them out to be a new layer but there are a set of tools in the toolbox that allow you to do all of this in a single step to search for extract you'll find tools such as extract by attribute extract by expression and let's use this one called extract by location i want to extract the weather stations that intersect the polyglots from this layer but i want to use the selected features only and this is a feature of the toolbox where each layer can be used as a whole or with the selected features so this worked and now i've got a new layer with only the weather stations falling within the wisconsin state you could see there is a warning here that says there is no spatial index and spatial mix allows you to speed up your operations where and it's very useful when working with large data sets if you see this warning you should definitely go ahead and run this tool called create spatial index it's a pretty simple tool you can run it on the points layer it's going to go ahead and create this page in the next video now i'm just going to remove this layer and run the same operation again a tip is that if you want to repeat uh some processing you did earlier or you made a mistake and you will go back and change the parameter and go to processing history and the toolbox mate is a history of all your tool runs so i can just click double click on this and it's going to populate the store with the same settings that i had earlier really convenient and now when i run it you can see the warning is caught and the tool ran a little faster so now i have the weather stations in the state of wisconsin let's bring in some weather observations i have those uh one as one file first station and the data looks something like this where you have a station id and for each day there is a data value which is the amount of snowfall on that day and this is the data for january 2020 and since the data comes in as one file per station i have over 400 files here it'll be nice to bring them in as a single table so you can use this tool called merge vector layers the tabular data is also a vector list so you can use this tool on them you can bring them in directly into this tool via this add directory input and now you can see i have 424 tables that would be merged and brought it as a single table really useful when you're working with large data that you don't want to bring into qgis first and then run possibly you can run them directly you can browse to them directly from the tools and bring them in okay so now i have this table i have for each weather station id i have now snowfall measurement for each day let's say i want to compute the total snowfall for each station in january i can sum up this data value and group them by the station ids to get that result so this is the kind of combinations called group stats and in the earlier version of qgi's you had to use a plugin for group stats to do this not anymore if you just search for group stats you'll find this tool called statistics by categories and this also shows that you don't need to remember the tool name just search for what you want to do and you'll find it so two stats on this field called data value and then group them by the station ids the result of this would be a table where for each station id i'll get various stats on this one particular field so the sum field here contains the total snowfall for the month of january now we can just do a table join with the points and i'll get the spatial layer with the total snowfall uh spatial joins can be done using this uh you can search for join you'll find this multiple tools you have we can do join attributes by field value where you can do a table join you can also do spatial join using the join attribute by location uh there's an a fairly new tool called join attribute here is where if that is the layers that you want to join don't intersect but you want to pick the nearest feature from another layer this tool is really helpful let's just go into a table join i'm going to join my extracted layer with the newly created group stats there and i just want to pick some field and i've got a special layer where for each station i've got the total amount of snowfall in millimeters so i'm almost done but my attribute table looks kind of a mess it'll be nice to organize and make it a little neater so that brings me to one of my favorite tools if you take away anything from this talk is that you can use this tool called refactor fields this is a really powerful tool it allows you to add delete reorder and even run some computations and feed all in a single operation you can select and delete the columns that you don't need you can even reorder them you can rename this maybe let's rename this column to total snow you can also apply expression on the source data so my source data contains values in millimeters maybe you want to divide it and convert to inches so now let's save it out to an output file just control this concept so now i have this really nice uh attribute table where i have the weather stations the names and the totals of all images right cool moving on the next example that i want to show so the next tool i'm going to show is called import geotype photos this allows you to bring you your photos with electron information as points during utilize nowadays if you take photos from your phone and your gps is on the light long information is captured as part of the photo metadata if you used for apps such as open camera you can also capture the view direction the compass direction along with the light lock here are some photos i've taken and this contains the light log as well as the azimuth information and qgs can read that directory of photos and i'll get an output there which is the points there you can see i've got the light long as well as the collection field let me add a background map so you can see the data more clearly there's a tool called create wedge buffers that allows you to do a nice visualization out of this data let me set the azimuth to be the direction field and create offer like this so this allows you to show your data in an interesting way and also useful to see which areas were covered and you get this nice view code from your data another tool you can look at is if you have trajectory data or movement data and such as say bird movement data or gps points and if you want to create paths from it or lines from it earlier you had to use some plugins there is a tool called points to find that allows you to take a points layer have an order field such as timestamp and you get a line here as an output pretty useful to if you work with some limit data let's do a fun one you might think of processing toolbox as meant for only the jio processing tools but you can see there's a cartography tool and there's an interesting algorithm called topological coloring what this tool does is that it given a polygon layer it's going to allow you to color it so that no adjacent polygons have the same color the way it works is it adds a field called color id to your uh input layer then you can use this categorized symbology to color your map so i have a new field called color id that was added by the tool and now i can color my map using that and i have a map of the us with all the states colored using five colors and more states have the same color all right so we so far covered the tools that we've used from the toolbox but some of the tools are really useful to be used in a model moving on the next tool i want to show you is a tool called download file this is a pretty simple tool allows you to input an url maybe to an ftp site or an internet site or a web http site and it allows you to download this data to your disk so let's say you want to download this data from first which uh produces active uh map effective files and this is a constantly updating file so every time i download i get the fires in the last 24 hours just copy this link and download this save it to a file it's going to bring that file in and download it and i got this points there now you may say this is not very efficient i could do this as easily or even easier using my browser but what this allows you to do is that now you can use the studio in your model and automate this process so let me show you this model that i built which downloads the file from the same firm's site it also uh uses the base layer such as a map of the u.s states layer and clips it to that boundary and also applies the styles there's a tool called set layer style so let's run this student what happens i'm not doing anything the tool is going to download this data flip it style it and add it to my map so you can see i got same data set but now it's clipped to the base layer and also nicely styled so this is really helpful when you need to download and process the data from the same url or even if the url is changing slightly you can set up an expression so that every day when you run this tool it fetches the right data set next i want to show an example of a custom tool say i want to take this whole workflow a step further i want to automatically download the data processor to style it and also save this out as a map so i wrote a tool which allows you to take your current map view and render it to an app template so let's see how i can use this along with the tool here so the tool is called export layout is pdf and i have a blog post that shows exactly how to do this and i can use a template file i have a map template here i can set a custom title [Music] um let's see what happens when i run this so my model is going to download the latest data process it apply style file and then render a map template out of it so here you go i have a pdf file that was created completely by the model and this is really useful to automate some of the workflows like that the last two that i want to show it comes from a plugin there's a plugin called ors tools which allows you to do network analysis using the opened out service api and this api uh uses the open statement data and allows you to do a bunch of really sophisticated networking analysis using the data so let me so this data set is the the bart station locations in san francisco this is the light rail station say you want to figure out where to stay in san francisco so that you are within 15 minutes of walking distance between them and i don't want a plane buffer i want to compute the actual distance that uh along the road network so i can use this tool called isochrones from here and set my mode to foot walking and compute the 15 minutes experience and you can see there's no data download happening all the data is already in the server the servers will compute uh the isotherms based on this request you just get the results really useful you can see uh these areas are all within 15 weeks of walking distance between the station it's interesting it's picking up the ferry routes as well all right so that's it i know there are a whole bunch of tools but this was just a preview of some of the tools that you can use and definitely explore the processing toolbox more i learned from it every day just search and you'll find some tool that helps you know analysis i can take questions down how are you javel um one question that came up is uh about the conditional branching tool in the graphical modeler do you have any examples of that or can you describe it uh yeah so this is a kind of long-awaited tool that was released for 314 and what it does is like you have a model where you can have a condition where on one condition output whether it's true you do a different processing on the output is different you do another processing so there's a tool called conditional branching in the processing toolbox that you can use in the modeler and the lawsuit i don't have an example right now but i believe in the the change log there was an example that showed how to set this up but it's fairly new and i think it enables a lot of cases where in earlier days you have to you know build multiple models run them in sequence here you can just run have a single model that can do different things based on the output of your result great you also um at the beginning uh mentioned quickly a plug-in to install whitebox tools in the toolbox um can you explain what whitebox tools are and what's in there for people that aren't familiar yeah so this has been uh this is an open source project uh similar to grass and saga where this collection of tools and you install it you download a binary and it works it's a cross-platform uh toolbox and you install a plug-in for qgis which gives you access to run those tools from the toolbox and they have over 400 tools and it's all open source and really powerful to life they have uh image processing tools and linear processing tools etc and i find them you know uh when doing image processing or some of the operation i find them faster they're all kind of multi-threaded as well so i find them faster and better structured than say grass or saga so i've been kind of using them and uh it's really great it's still in beta it's not it is uh fully stable yet but you definitely install it and i've used things like say i wanted to view shed computer view shed and there's a nice you shed to where you have multiple points and creates a new shade out of that and last week i was exploring histogram matching between satellite imagery and their instagram matching tool there so really nice uh collection of tools which the qgis doesn't really have many raster tools in the default toolbox and this adds those as well so uh definitely i'm pretty impressed at this collection and pretty excited when it actually becomes stable and released and i think we have time for one or two more we have a question um about the export to pdf how is the the pdf tool different from export to pdf uh so the export to pdf works if you are in uh print layouts so there you create a map and export your map so if you wanted to do this automatically you can't go and click this button so this tool uses the same apis but it's a python code that calls that export function from the processing toolbox and the idea for this model that i showed was that if you had to do this every day and i knew somebody who had this job like every morning to come download some weather data join it with some of the internal tables and spend an hour creating this map and pushing it to like the internet site and you can now do this in a just click of a button where you download the data and everything and your map gets generated and you can even push it to uh some other place so all this is automated it helps you save time and it's less error prone etc so this was an example of kind of using the same functionality from a processing toolbox model fantastic so i think um we're just about out of time and i just was wondering is there anything else that you wanted to add at the end of your talk no no thanks for uh the opportunity here i would like to remind folks if you enjoyed this talk and if you've been enjoying this conference do consider donating to the qgs project this is a free conference and a lot of effort is called by the community to organize and run this conference and uh if you can just visit qgs.org and you know give some monetary initiative whatever you can afford i think it will be really great for the project and uh would appreciate that so that's it thank you everybody thanks so much you double

Channel: QGIS North America

Views: 3,041

Keywords:

Id: Tkqj3VCxrqI

Length: 22min 0sec (1320 seconds)

Published: Fri Jul 31 2020