Advanced QGIS Expressions

and today i'll be talking about advanced qgis expressions if you follow me on twitter or have been following my blog you know lately i've been really obsessed with some of the advanced qgs expressions and since i've learned of them i've discovered that they can be applied to a whole range of applications so today i'll talk about some of those learnings a brief intro about me many of you may have seen my tutorials at qgis tutorials.com i've been writing and maintaining those for over a year now i recently started my own company called spatial thoughts which is completely focused on training we are one of the qgi certifying organizations so we offer online qgi certification training and i also do python for spatial analysis and google attention trainings i'm also a visiting researcher at university of journalism where i teach qgis and urban planning a brief note this talk will cover some of the advanced expression techniques and if you're not familiar with the expression engine i highly recommend this class that niall dawson did where he covers futures expressions variables and data defined overrides it's a really nice broad intro which will give you a good understanding of how the expression engine is set up in qgis in this talk i'm going to cover these three functions and you may say just three functions for the 20 minute talk but believe me these are really powerful functions and we'll explore this in detail and i'll go through some of the examples and use cases for each of them so let's start with this aggregate function i want to tell you a little bit of story of how i got introduced to this function last year i went to the qgis user conference in spain and during one of the talks i was sitting with lynn fisher at the back everybody knows lane she's a professor in denmark and after the talk we were just chatting and she asked if i knew of a way to automatically fill attributes of a vector layer based on how it intersects with another layer and we're not talking about just doing a spatial join we wanted more of a real-time application where as you draw the polygon you can autofill some of the attributes based on where the newly digitized polygon intersects with other layers so i didn't know of how to do it at that point because the way i used expression engine is with a single layer where you have a layer and each feature has some attributes and geometry and when you write an expression say you are computing a new field the expression can use the values in each of the attribute fields or you can even use the geometry of that feature and as the expression gets computed for new features they use that features geometry and attributes so there is no way to kind of look up values from other features when you're using this kind of functions but then i slowly discovered that there is a function called aggregate which was been there around in qgis for a while but it was not really explored with there were a few blog posts and few stack experiences but nobody has really applied into the situation so i looked at this function and seems really interesting because when you write an aggregate function you not only have access to that layer but you also can have access to features from another layer so in this case we have our layer here we are writing the green expression here but that expression can also look at features from another layer and as the feature the expression gets evaluated it can look at the attributes for the feature that it's being evaluated for and also all the features from another layer and you can you do all the computation using these features as the expression gets evaluated further features they get access to that features attribute and geometry along with the features and attributes of all the features in the layer this is quite powerful and you can apply this in a whole lot of different settings now one thing to note when you're writing the greek expression you can refer to the current feature that is being evaluated using this variable called addparent and you can refer to the attributes and geometry of the layer 2 just like you would in any expression so let's see how the indent function is set up this function takes six parameters the three of them are mandatory others are optional the first one is the layer name remember you are writing the function here in layer one and you can specify the layer name of the layer 2. so when you're writing the function you can say i want to read features from another layer then you can write an expression that once the function finds the feature from another layer what do you do what with that so you write your expression there you can specify an aggregate type and this could be more of a statistical function such as count or sum where you can say look at this other features and find the count of them or find some of them you also can write use functions such as concatenate which gives you all the values from another layer and you can just get a comma separate string out of it or you can get an array which is a list of values you can apply a filter that means you don't want to look at all the features from the layer 2 but you want to want to look at the subset this is where a lot of power comes from where you can say look at only the features that are intersecting with the current feature and then you can do some computation on that and there's an optional order by parameter where when you are doing concatenate or array you get a list of values and you can sort them in using another expression if this is best done as a demo so let me just demo uh the solution that i figured out to lane's question this is qgis 314 and i have got some data from city of san francisco this is the parcels layer and you can see that there is a parcel id that i've labeled each feature with there is a polygon layer which is currently empty and it's got three fields fid count and passes and what i want to do is as i digitize new features in this polygon layer i want to look up the intersecting features from the parses layer and automatically fill this parses uh attribute with the the list of intersecting pulses in qgis you can set up custom attribute forms for each feature as you are digitizing and here i have set up the attribute form with the default value where that's where i've written the expression function aggregate expression so when the new feature is digitized and the attribute form pops up this will be the default value based on this expression the expression says read features from the parses layer just get their fid the feature id but look at only the features that are intersecting with the currently digitized features and then give a concatenated string of all the feature ids let's see how this works in action so now i'm drawing a polygon so here i draw this new polygon as i finish my polygon the aggregate expression gets evaluated and the resulting value is put into this field called parcels you can see it has populated this field with these two parses it's intersecting from here you can also draw a larger polygon which is going to do the same for all the recycling parts you can see this would be a pain to do this manually and you can not only populate this automatically but one of the interesting applications of this is in data validation where in this case say we had a zoning layer and as you're digitizing new parcels you could have some constraints where if the newly digitized polygon falls into certain zone then other values could be within this range and so this kind of real-time validation could be performed using the elliott expression i have a few more examples on my blog you can check out and see how you can apply this kind of techniques to help your digitizing and data editing operations so i clear expressions are great for digitizing and data editing but there are several other use cases an example of such an operation is this problem of finding neighboring polygons this is one of the more popular blog posts on my blog and the problem is that given a polygon layer i want to know for each polygon which are the neighboring polygons one of the key insights i had was that you can use aggregate function on the same layer it was designed to work with two layers but rather than giving a separate layer for layer two you can say do an aggregation on the same layer and this allows us to take each feature and compare it against all the features in the same layer and then we can find which pieces intersect and find the movies let's do another demo so here i have the layer containing all the us states they are labeled with this attribute called st usps which contain the short code for the state name and then for each state i want to compute the neighbors let's see how we can do that it's going to go back to field calculator here i want to have a new field called numbers to be a string and pull up my saved expression here you can see there's only one layer i'm applying the function on the states layer and the identity is also using the same layer and for each feature it compares itself with the intersecting features and gives me a concatenated string of all the intersecting states but you can see for each state it also picks up the state itself for alabama i also get this al here which is the same state because it's intersecting with itself so let's see how we can remove this i want to switch from this concatenate to array which will give me an array as the output and then i can apply this function called array remove all and remove the the feature from the list object so now i have a list of all the neighbors excluding the feature itself and i can convert it back to a string so you can use this array to string function and run this so now it's going to compute for all 50 states evaluated against all other 50 states and find all the features that intersect each feature and compute that the neighbor string let's have a look so you can see now for every state i have a list of neighbors here and you can see california neighbors are nevada island oregon and so on and so forth really useful and this would have taken a fairly complex python script but you could just do it with a single aggregate expression now let's look at this other function called array for each this is one of my new favorites because it allows you to run for loops within the expression engine you saw from the previous example that you often get a list of values and what this function does it allows you to apply an expression to each of them and compute a new list the syntax is fairly simple you give it a list and then compute an expression so in this example we can given a list of numbers one two three you can refer to each item in the list with this variable call element and write an expression such as element plus one so the result of this expression would be a list two three and four so let's take our previous problem one step further say you have the neighbors now but you want to compute the shared border length between its neighbors so for each state i not only want to compute which are the neighboring state but what is the length of shared borders between each of them so this is quite a complex problem where we can solve this by first converting our polygons to lines and then computing the intersection between each neighbors and then computing line so let's see a demo we'll keep working on the previous example the first step i want to do is get a line layer out of this and there is another thing called boundary which allows to get a polygon boundary as a line feature so i'm going to run the sun okay once i have this layer i can now get on view calculator i want to add a new field type string this expression this looks quite complex but what i'm doing here is that you're looking at the neighbors attribute that we did previously and looking at all the neighbors and for each neighbor so this is where the array forage comes in handy you are computing the intersection of that with all its neighbors and computing the length of it and converting it to miles so let's run and see the output here again this is a quite a complex computation but you are able to do this with just a single aggregate expression let's look at the result now i not only have the neighbors but i also have their border lengths so you can see here that california shares a 613 miles border with navarna and 235 arizona and so forth really useful and handy the last function i want to cover is called eval what eval does is that it takes a string and evaluates that as an expression what that means is that now you can use the expression engine to construct expressions we are into full inception mode we are multiple levels deep i'll try to explain this with the demo you say where you would want to use this one concrete example that i found recently when i was working some weather data if you work with sensor data the data is often collected on a periodic basis so you have a weather station which is collecting data every hour or every day and then the data is then shared in a format which is not ideal for gis something like you know you have 24 columns for each out of the day for each station and then you know if you want to compute the total for the day you need to sum up this 24 point let's do a demo which will hopefully make it clear i have this table containing some weather observations let's have a look the there is a weather station id and then this data is for the month of january 2020. and the way the data is structured this i have columns like value one value to value three and this is for each day so value three means the third day of that month and so on so i have 31 columns for each day of the month where the amount of snowfall is captured say i want to compute a new column which is the sum of all the days so let's see how we can do that so i want to add a new column called total now you could do it in a simple way and just pick up the fields the value one plus two plus value three and so forth this works you need to do it 31 times it can get quite painful if you add much more columns so let's see how we can use expression engine to help us build this expression so you can start with this function called generate series which allows you to generate a sequence of numbers i'll generate the series from 1 to 31 you can see here got a list of numbers from 1 to 31 then we can use our handy array for each function and compute the string so i'm going to compute a string with combining the value and element of the list so now i've got a list which says value 1 to value 31. i need to have the values in double quotes which refers to the column values so i'm just going to update this so now i have a list of columns value 1 to value 31 as a list i can create a string using this function called array tool string but instead of using comma as a separator i can use plus and now i've got this long looking string which works as an expression right if i just give this to an expression and you just say i think you mean asked me to sum up all these column values and you get the value but we already use it for expression engine here so how do we pass on the results of this to the expression engine again that's where eval comes in handy so i'm going to pass this on to eval just going to evaluate the result and you get the result here so let's look at the result now the end of the table i have total column which is the total of all the 13 values really useful in a lot of ways where you can use expression engine to construct expression itself that's it that's my talk if you want to learn more about expression there's a really excellent documentation it's a fairly new uh where it's on a single page you have documentation about all the qgis expression really useful and if you liked this talk uh come to my blog there are a lot more fuji i suppose there i am open for questions now this that was fantastic so um one question we got on the the chat is has to do with the um with variable function do you have any examples of how you might use that uh yeah that's a great question this is uh i often wonder because you're writing these long expressions and when you're programming you have access to variables where you compute something store it in a variable and then use that variable again so you don't have to keep defining this long things so yeah there's a function called with variable where you can say with variable x do this and you can define this long expression there and then the value is stored in this x and in the next parameter you can use x and will be replaced by the the value that you define so i used to use like you know i had some use cases before earlier uh there were some functions that were missing so i had to kind of compute the same thing over and over again and it's useful there i don't find myself using too much of that nowadays uh i always find that there is some built-in uh parameter that allows you to kind of reuse some of the values computing one example was uh order by uh the earlier version of qgis you have this aggregate expression for order by so you have to kind of do some fancy things where you compute this aggregate again and sort this and do this but now aggregate function takes care of it but if you have if you find yourself writing the same long expression again and again in the same expression uh you can use with variable uh function to do that great we had some questions too about um the aggregate function specifically if you've run into computational limits with that on or or data size limits where where it um becomes unwieldy yeah i think this was one of the things i learned the hard way that once i discovered it i said i don't need any processing tools i can do all my spatial analysis using aggregate functions now and i you know you know started using it and i quickly discovered that the function doesn't use spatial index yet hope mal dawson is listening so but uh so right now it's kind of uh you know if you have a thousand features in one layer and thousand features in another right negative expression it's gonna do a million computations right so it doesn't scale very well as much as a processing toolbox so on the tip is that if you have a processing tool that does this use that that'll be much faster because they'll use processing use patient index and it's just more optimal so yeah it's definitely not optimized uh but the advantage of this you can use it in a situation uh where you can't use anything else like you know if you want to label something using your expression or if you want to you know when digitizing would do something so there's no alternative where you can run some tool so it kind of fills in that gap but definitely if you're doing some complex spatial analysis it will be slow because it doesn't use spatial index here great and one last question for the eval function would you need to do field calculator again if you append a new table uh no it's a so the field the way processing algorithms work is that they compute a new layer out of it so when i apply this eval the new layer that got computed by the processing toolbox that is the static layers that the field values are the completed values it doesn't contain expression if you had for example a virtual field instead of a field like that so if you use the the other field calculator that's you know part of the or you just added a virtual field instead of a new field then that would just contain the expression and then that means that when you're using that again you can you know use that but that the field itself contains the full expression that needs to compute the value so i do need to run ebala here on that great well thank you very much you draw well that was fantastic thank you

Channel: QGIS North America

Views: 4,587

Keywords:

Id: IXPCec8vgLA

Length: 22min 40sec (1360 seconds)

Published: Fri Jul 31 2020