Edzer Pebesma: "New R packages for spatial and spatiotemporal vector and raster data"

thanks everyone for you know for voting me in on this on this program which apparently was very competitive as we saw I'm sort of I feel very much over represented by the amount of time that I take in in the problem in the program and somewhat somewhat embarrassed but I'll try to you know to sort of make make my way through it this talk I appreciated the title because this kind of this nice slight theme doesn't work well with with long titles it just looks awful so you need to have a title that is kind of thirty characters or doesn't work this talk is kind of a reworked version of a talk that I gave earlier this year on invitation from our studio cons so also thanks for them for sponsoring because that was sort of a large audience there were 1200 people and and 600 in the room or so and I sort of I made a much larger effort for for my presentation there than I you then they usually do that is very embarrassing but so so this is but anyway there's a couple of nice things in it so I will say something about new art spatial packages and now and then you know jump out and do other things to to make it more chaotic and and to try to keep you awake which is not you know which is not the easiest thing so spatial data are essentially everywhere as we know in this all kind of all kind of stuff that goes on where you see maps and and the thing is that everyone is now these days is now making maps of all kind of things and that is a kind of challenging right there was like two days ago there was this - tweet tweet from Julia zilch me making maps but not really knowing anything about maps projections right so this is I what I feel like every data scientist is it's nowadays kind of running into oh oh I have spatial data well let's make a map I have no clue what's going on let's see how this works and and and whatever goes somewhere you know worst case to Twitter and now she dearly ourselves is a pretty well-known person in the our community and the data scientists I think she's associated with Stack Overflow I'm not not entirely true but this sort of got a lot of responses from people actually giving advice he had some great some you know some initial ideas and so a lot of response came out of day so for her this helped in this of course doesn't help for everyone so I wouldn't sort of say that this is the way to get your help but sometimes it is it is funny and it's useful so I will say something a little bit about spatial data in are about simple features the simple features the SFR package that I've been working on most of my time last few years say something about you metrical operations there's a lot more going on about simple features tomorrow morning more in a tutorial style say something about the tidy FERS how things interact there is also a lot more about on that and in Robyn's talk I think on Wednesday or on Thursday but this is kind of give you an overview so the original title was kind of tidy spatial data analysis kind of sort of you know going towards the the the tidy first our studio community and I will say something about rosters array spatial data cubes and the Stars project so I will say actually very little about other people's packages right because there's a lot of people here talking about their packages that will get their time and and there's other things like packages new packages like team app order or the Google Waze package from from from Daniel Cooley I believe the you know they are very exciting but I know I know extremely little about them so it's not my my sort of I cannot say much about it so some personal context I studied physical geography I'm a geographer I did a PhD and essentially in spatial statistics and here statistics I now teach team for medics in Minister in Germany where I'm associated with the Institute for geo-informatics and I did a lot of programming so originally everything was in C this was in the 90s and later on as plus and so on things that were that Roger mentioned and wrote our road or contributed quite a bit to to several our packages in the early 2000s notably teesta SP space-time trajectories and a couple more and recently so I'm one of the authors of the applied spatial data analysis with our book that got two editions that we essentially wrote after after finishing the SP package in a couple of packages that could use these classes so we had a that's a coherent framework in and could publish that in 2008 and this is got a second edition in 2013 because a lot had changed and we were now essentially sort of again in this phase of finishing up work and might sort of start publishing book on that later on recently I developed an interest in working with measurement units and that I came into you basically by partly by because I inherited a course on spatial reference systems which I you know which I enjoy teaching but they had to learn a lot about sort of reference systems where they come from and their measurement units play play a role this is the this is sort of connected to what what Roger mentions is is this network analysis on packages on Chrome this is already a couple of years ago was tweeted by calling Gillespie who co-authored the book together with Robin Wright on efficient our programming and that shows sort of a couple of clusters in in the in the our packets Network and you see sort of things web services visualization interfaces to C++ maths which is kind of the a lot of sort of mix of applied statistics routines that came with the book from being Athena Paulson Ripley and there's a little SP cluster kind of that that is also not very large but very visible and interestingly DSP sort of told odd pick special packages that sort of arts of having built around that are the only ones that are that at least have some domain of course the domain of spatial has you know is again all kind of domains like Epidemiology and ecology and and biodiversity and soil and so on hydrology so there's all kind of domains but they're kind of a common umbrella of spatial stuff and it turned out that that works that works quite well kind of you could call it well the tea is umbrella I don't use the word tea is a lot of it maybe it works or our GU computation which they also don't don't use a lot but it's also appropriate what makes spatial data challenging so we don't have a clock right here so I know I don't see where we are in terms of time yeah so change the challenges in spatial data is that the earth isn't round at wheat so we have sort of an irregular pretty irregularly shaped planet in which we live and it's it's kind of from a distance it looks like a sphere and then we start modeling it we say it's a flattened sphere but in the end it's not and so it's not sort of trivial to measure this thing yet so we use a lot of latitude longitude but it's it's it's difficult to sort of to relate often one to another so called coordinates always consist of two or three numbers yeah and these coordinates really lose their meaning when considered individually so if we have like a location of this is location latitude longitude it's it's it's really a couple of values that that sort of denote that location we could say okay we put it in two columns one we call latitude the other we call longitude when we have points that's easy we have them there but they look the longitude values sort of are useless without a let acute values yeah so there's it's not like the typical other variables like whatever time of observation or something like that or ID or last name or so these these things work sort of individually but like longer you let you do not do not work so they really belong together and that is one and then the second problem is that the most common form is longitude and latitude and they're you know they're funny they're really sort of spherical coordinates and there's a lot of things you cannot do with them for instance if you compute distances and that's what a lot of people do compute do whatever cluster analysis or something like that or they just say oh well we do machine learning we put you know machine learners just need features we throw and let you to launch it and we'll be fine right and my feeling is always that we're not entirely fine right because they are slightly different from from other features in the sense that they have this property that that numbers belong to get it in just not sort of single features but people do it and well if you're successful of course you could argue what what beats it right why wouldn't you do it and so but in any case the the thing is that if you would do if you would look at distances between points and used the standard Euclidean distance routine starts test then you don't get distances and you know unless you're close to the equator where it kind of is sort of linear related to distances but otherwise it's not and you know you can then of course scale and do stuff but then you're already sort of working with with projected data your challenge is that as I said the earth is like sphere like a maps and screens are flat so anything we see is it's essentially is 2d right our retina is 2d so sometimes we get an impression right on the first screen I had is moving this moving globe thing so which gives an impression of something 3d but anything we see constantly is 2 days or anything we see is what we call project the data and there's all kind of projections you can take and they have properties and if advantages/disadvantages project the distance are always distorted yeah so there's no way of projecting and retaining distances it just doesn't there's not possible it's like you know making it an orange peel flat you have to sort of break it up and so on this doesn't work so you have to choose things like preserve shapes preserve directions preserve areas and so then you can you can you can do stuff but one goes at the cost of the other essentially so and then there's sort of a last very nice last thing that even if you have long you let it you values coordinates thing with your latitudes then it's not even unambiguous what what they actually mean because they're related to to a date and they are related to a model of the earth and they're different models of the earth right and that sort of matters not always a whole lot the right as I said here the item is unlikely important when you map continents yeah but it is when you try to deliver pizza with a drone or something like it so if you're sort of doing things where distance of tens or 100 meters or something like that matter then datums become important so you need to keep track of datums yeah so if you have loaded long acute latitudes it's not like like you're there you know it's a fear you know enough or something like that there is you know there's this tendency that these days that everything should be wgs84 which is the sort of a global datum but it's it's not you know it's only a partial solution and it's not it's not likely a problem that's going to go away yes there is things like continental drift and so on so so so you know the earth it's not like a static thing and so it's made it's it's doable but it's it work yet so you need you to put effort in there now there is this thing called a simple feature so the little history is sort of weeds so that we came up with the ESPE package like 10 or 15 years ago yes the idea came up 15 years ago on first version 13 years ago with just points lines polygons and and some you know some efforts to do grades or rosters and it's sort of during that same time there is this there waist is let's say it is the standard in whatever the the standardization world sort of coming up that talked about simple features that's a way we can represent things slightly differently and maybe more consistently by like by Columbian some features and that comes from towards sort of the word feature the feature means that it's a thing it's an abstraction of a real world phenomena so it can be a type or an instance but not won't talk about that but it's in any case it's a thing that has a geometry yeah so like this table you know has a geometry it means that it is somewhere it has location but it also has like a shape and so we could describe that shape and so and then another thing is that it has kind of properties like you know it's made of wood and so long it is steel feet and so so we could we could describe that there's a part of it is the geometry and part of it is this other properties it has a weight in these kind of things now how are we going to describe that and then we come to simple features that we say okay we're gonna sort of describe geometries by sequences of points that we connect through straight lines that makes the thing simple essentially yeah so no curse that's it and then we can deal with that we can do points we can do lines sets of lines we can do polygons polygons with holes and these kind of things and and that's sort of a way to represent these things so yeah so we represent geometries and simple features by points lines or polygons or by collections of them yeah so you could have like a geometry collection that consists of line in a polygon we could why that is we get to that later so this is a this has been described in documents that you can find online that are that are for standard documents actually reasonably readable I would not sort of dis recommend anyone to read these standard documents they're they're kind of kind of insightful and and relatively relatively well written so you so I saw is an organization where these documents and the way you have to have to buy them for like hundred fifty Swiss franc or something like that but otc open to you spatial consortium is the place where you can find the documents really online in essentially their identical form just not with the ISO label on them so this is relatively friendly the main thing is not that it's sort of written down in standard document but there is a lot of software that adopts these standards and software in particular libraries that have been developed in that say that are now under the umbrella of the OST or the open source to use spatial community tos and data library notably and a couple of other ones it's adopted by do Jason which is a standard for map information on the web so so it does simple features but then even much more a little bit more concept - also saying that their coordinates should always be longitude latitude and and should always be wgs84 so the tube chase and even much more constraints things tu Sparkle is a sort of a dialect that uses Sparkle sort of linked linked data dialect it uses simple features to describe geometries it has a couple of a couple of encodings well note text and well-known binary and that means encoding is really means there are let's say standardized ways of expressing how for instance a point or a line string is is written down that is sort of system independent so if you have different systems if I'm going if I'm in R and I'm going to communicate with Dida or with TOS or with a post GIS database I can use that form of encoding my data to read and write so it's a communication layer really it's an encoding and that makes the thing very flexible and powerful w WK base well non-binary is really the the form used by by pretty much all spatial databases of post GIS MariaDB which used to be my sequel I think SQLite and and all sort of pretty much all the spatial databases put put sort of well-known binary binary blobs in blobs big large binary binary large objects and and it's there right so it's just raw binary data that that you can read and write when you when you can read and write these these encodings the standard this simple feature standard not only sort of describes how to represent this data point inés polygons in this way but also describes a lot of topological metrics predicates and operations right so friends you want to be able to compute an area of a polygon you want to be able to compute the length of a line you want to see if these sort of you know points are not completely messy and sort of mixed up and then do checks so operations there are right are for instance logical predicates like is a polygon valid is a important simple is a polygon empty quantities that you want to compute like length and area dimensions that you wanna insert of compute from you know is it's a point or a set of points then it's zero dimensional is it the linear thing then it's a multi and it's a line string or a multi-line string or it's the surface like a polygon or a multi polygon and you can also derive geometries so you can do buffer centroids boundaries convex hulls simplify things liners so all these kind of all these operations that we kind of know from from GIS and that whenever you've worked with post GIS you've run into them and then probably grass uses the same terminology hopefully they are the kind of these words basically kind of described in the standards like buffer should do this buffer should do that convex host you do that simplifies you do that not not not really all of them but but kind of this is how it works there is sort of these are operations single things these are operations on pairs so you can four pairs of four pairs of geometries you can compute prints the distance between two points or the distance between two polygons you can do predicates you can see whether two geometries intersect right meaning that a point is in a polygon or that two two polygons overlap or touch each other and so on and there's all these kind of constraints that basically build on the d9 I am the dimensionally extended line intersection mala which is a powerful model that that if you have timed and I'm sort of I'll talk a little bit more about it tomorrow probably so this thing about being simple and so on looks like you know looks like it's a bit of a crazy thing but it is whenever you can you can think of things that are not simple and I defined here a line string so this is basically this says I'm loading labor I'm loading SF and then it says I'm linking to a couple of libraries that I need if I sort of created line string here then easy oh it's it's a it's there so it gets printed directly and the way it's printed essentially this well-known text form right so the by printing it this way I could sort of send that directly to a database you wouldn't do it you would use binary but but anyway that would that would work out you could do this to do sparkle or something like that but then if you look at it this is zero zero one one zero one one zero so the sequence of four points that form in a line string if I plot that thing like I did here then I'm plot sort of the points a connected with lines you see that this is a line string consisting of four points of a line that that self intersects yeah so it's not so and that is a problem because if I want to sort of find out is this thing simple then it will say no it's just it is not simple yeah so that's a problem it's not a simple feature so I can sort of come up with a line string of these four points I can define it but later on if I'm going to use this line string you know in operations then it's pretty likely that the whole thing will break down somewhere with an error that says there is some kind of geometry that you're feeding me which is not simple I'm not going to do this is not a simple feature so I can technically construct it upon construction this check is not carried out and it's not sort of doesn't break but it will break it will break later on and then if it breaks then you can use the SST simple function to to essentially see if this is simple if I would have done if I would have done it the other way around sort of go this way and then yeah then it would be true yeah let's plot it it's the holidays so this you know oh whoops doesn't oh I should remove these simple yeah so you can see here that this is not really what I had in mind yeah so if I go the other way so long and then and this thing is kind of you know this is the line string that is simple it doesn't self intersect and so let's see if that it's true is simple then we see that this is simple yeah so this is this is true yeah so this is a predicate this is a binary predicate it says yes this thing is it's simple and the same thing is with valid with valid it looks it does checks for polygons and there is a lot of you know anyone who has worked with GIS sort of knows that there's a lot of polygons data that are they're messy that are you know if you would have like a multi polygon that consists of two rings and the two rings overlap or something like that or you have like if you have a polygon with a hole in it and the whole polygon sort of halfway overlaps with the outer ring that things shouldn't happen but you can't check that all the time there's in it's valid function that that does check these kind of things so these are kind of household maintenance things and then you can you can define you can sort of compute new geometries by for instance by looking at the intersection of of two things the difference or the symmetric difference and the union of things yeah so that gives all logical operations but that they kind of they kind of give the resulting geometries a two important special cases that that that the simple features framework has that the ESP framework for instance does not cope with and those are default the MT geometries and the geometry collection and here's an example where they come from so if I on the left hand side you see the red and the black polygon and then trying to compute their intersection and as you can see they don't intersect right but if I want to compute that intersection what should come out right so the thing that comes out is an empty geometry and now this is this happens to be an empty polygon and so they're they're typed empty UF's have empty point you have the empty line string and so on and it's completely to me completely unclear why we have typed empty things but it is extremely useful that there are empty things that we have basically it you could think of it as the missing geometry right the missing value for geometry and that is that is different from like having a coordinate with a missing value you could also think well coordinates and numbers couldn't they be missing that it's a completely different thing yeah that is like oh I don't know the coordinates dude I don't have long you yeah this could happen right if you only know like how high the Sun comes you know one of these two things but this is different right this is kind of says okay I want to do this logical operations of the two things yeah and there is no intersection yeah how do i encode it it's it's empty okay it's empty so it's useful the other thing is when you get do something like an intersection and you get you get a mixed back back right so these two right hand side black and black and red polygons kind of intersect in an area and in a line and in a point right so we have a combination of a point a line string and a polygon and that combination is the intersection of these two polygons yeah so it's a mixed bag you have to live with it it this is the way it is right so it's useful that you can represent it and typically it's it's bad news oh I have geometry collections yeah so you typically you want one of them yeah you want either the points or all the polygons so you're going to throw away the rest of them but first of all is sort of the question what is the intersection well it's a mixed bag so there's room for this for this infrastructure that is that it's important to have and that makes the whole sort of framework very very consistent and you can see that there that you know that in the in the standard document and simple features that there were mathematicians involved people who understood like set theory and so on that's a they said let's make let's try to make a framework that is that is complete yes and and there is only you know a few glitches but otherwise it's it's okay so packages have simple features does a lot of these things and it stores simple feature geometries as a list column and that's sort of to many of you that will probably sort of sound like complete abracadabra right which is which is fine let's let's try to to see what what we can do if it does that in objects of class SF that essentially extend data frames right and that also might sound very much abracadabra and so let's see what that what that means so let's see what a what a data frame what a data frame is just for five minutes because because we have some time so so who of you knows what a data frame is good good can I can I ask you to explain what a data frame is yes you you said I know what a data frame is yeah yeah okay thanks so there were a couple of keywords it's XY it's matrix like and it's table like right so who can explain me what the difference between a data frame and the matrix is yeah yeah yeah right so data frame can have different types different types in what yeah but like in different types of rows columns will have different types right so they're they're like matrices they rectangular the tables and columns can have different types yeah that is the sort of the the real sort of property of data frames and if you look at them what I what I do it's well it's it's always like where to start right so if I create a data frame let's say of the data set with variable a that is values 1 2 3 and a value B that has some some kind of character data you get this data frame right and so this is what you see this is basically the print method for a data frame so you see a rectangular table you see the column names yes and you see the row names they are default and then you see the values of these of these table things right so and indeed so we have columns with different types and what sort of what data frames what what are does basically is that it that it stores these things in a list yeah so we can ask is this a list yes this is true so who knows what a list is in our so I can explain me what a list is we see much less fingers Robin right so it's a series of objects they are stored in in a list right yeah so it's an object yeah so the general sort of thing we think of these objects as is what they are is a vector right at least this special type of vector that can have any types of things in it right and so they so in our it's useful to to understand that vectors are essentially everywhere so I defined here two vectors here it's a range from 1 to 3 and here as a as a set of 3 strings so these are all vectors and but they're of a single type right they're of character type array of numerical type and so on and list is that the generic factor that can have anything that sort of collects anything so list is a vector that has some length in that and so data frames are lists right but data frames are specialists in the sense that we could you know we could sort of look into the list but looking at the list extraction operator sort of you know give me give me element a of these lists yeah and I get this element a or you can say give me Alan B of this list and I get this which is then the character factor turned into a factor but that's another story so why I am Telling is the thing is that data frames are essentially column stores yes column stores is it sort of a database an ID and database so if you are familiar with databases relational database is typically our record stores right so they put you you say well this is how a record looks like record is a first name last name a each and so income everything and then you put a lot of these records in the database and so it's collections of records whereas in our we as essentially we work typically with tables in terms of collections of columns this is sort of what a dataframe really really is it's a collection of columns and the thing is that these columns have to be you have to have the same length if I'm going to try to define a data frame with different length arguments then it will sort of complain yes it's not going to this is not going to happen it says different number of rows this generates an error I cannot be kind of sort of combine these I can of course combine these into a list but it's not like this will be very useful for doing analysis or something like that because we would argue how does the third record of this list look if this is if we think of of data frames as things that have records then really we are there right so we think of this yeah is this the third record two rows we think of records and columns with sort of the list elements a and B we think of as columns right so yeah so then data frames have a couple of other things they have so called attributes like a lot of sort of so most our objects are vectors of some kind yeah your numerical character or their lists like generic factors that have elements in them and they may have have they may may have attributes and attributes a kind of the metadata we think of them have metadata and so if there are attributes any attributes are names the column names two data frames will always have named columns if you don't give it names it will create names like here if I say well just create this thing from from this yeah then it will sort of create default names which are Terry but pretty much terrible it creates them tries to create them from the arguments but it's it's named yet it will be a name it has a class the class is data frame and it has a row names attribute which is essentially the the row names which are given here by default I could override it but there is nothing that that sort of with halt you to corrupting these things for instance I could now easily say well you know I'm going to I'm going to change this this row names attribute into something that is you know doesn't have length three but has length to right and then I have then I have like a data frame which has which you know which has a corrupt row names because row names in L length to where where the variables have length three so let's see what happens ah so something is printed but I get a warning message here I have a corrupted data frame so the print method here does a check for me yeah but nothing sort of I'm sorry ah because if this is like oh I see its certain if the screen is too large okay here it was can use this like I'll see if it can make this screen good yeah so we you know you can so it's just an object right with a class and with has properties and so on I can sort of I can go along and and and and do all the things that that you know that that we know that shouldn't be data frames and then do things with it but I don't want to do that but it's it's interesting that well this is where basically where s4 came in and said we need to have more formal classes and so on but it's still interesting that right now essentially the s3 classes like data frames are still kind of ruling stuff and all day they are fairly unsafe and it's unclear what a contract with your API really is it's it's it's working yeah then everyone uses it now we're coming to list columns yeah so we have here two simple columns one is numeric a and B is of type factor or character doesn't really matter so what if I sort of add to that something that is a list column and it has a list of length three for instance with some irregular data yeah then I can see ah doesn't like that no it doesn't like that but it's really the data frame function that doesn't like that that says this is sort of too messy what you're now providing me but I can easily add a variable called list column that has this it's element here to this data frame and then and then have this thing yeah and what you now have is essentially a data frame with the columns a and B that we already knew and a third column in it that is not of a single simple type with that is essentially a list that has the right length but there is kind of a messy type right it has differing number of elements yeah so they could be like pointers to records to which I have a connection or something right you could sort of put a graph sort of it's sort of a graph structure in this kind of this kind of thing and that makes it very that makes it very powerful that that you can sort of that columns don't just have to be of a simple type yeah numeric factor character time or something like that they can be of more complex type and that is yes it's exactly what we do with this list column thing in simple features and then we sort of put more complicated things into you list columns things like multi points like line strings like polygons Multi polygons gm3 collections anything we just make sure that the list column has the right length yeah and that we put it in a data frame and we do it we can do the same thing with tables essentially how does that work well this is what you see here we see a print out basically a table print out of tables tables have slightly nicer print abilities than data frames they put a they put a your studio team put a lot of effort in doing this although some people will not like you know this kind of little things that less significant digits get other colors and so but it's it's good fun here you see that we have a table with four regular columns and when geometry column width points this is the Meuse data set we see the geometry is essentially put in a list column where you can see that the type is your point and the units are meters and here we see the North Carolina dataset where we have multi polygons with degrees as units in in certain lists column in a table yeah data frame does does essentially does the same thing so this is the this is like what we call an SF object that knows that I that it has attributes but it also has a list column with a geometry and it end that the geometry column has a particular name that doesn't have to always be the same thing so we see the geometry list column essentially as a column in a data frame and it has a particular class it's of class SFC I don't know why as simple feature geometry collection or sort of set of geometries they are here and a single element of that is then again a class thing which is a feature of geometry which is an object of class SFG a simple feature geometry so the list column is a collection of these sft object really and the feature that as we think of is then a record in this data frame or in this table so that has a geometry and it has a set of attributes but oh of course it's not the way it's stored because it's a column store type of thing yeah but we think of records here as as features they have properties at non geometrical properties and geometrical properties right so this is kind of is the framework that SF uses and that is so it is really they these objects are really data frames even if their tables they claim to be data frames you could argue whether they are but people say they are so we'll believe that good so that is that's an important difference from sort of the whole SP framework which where we had objects that kind of behave like data frames but that are not data frames and that turns out that that turns out to be the critical thing for data to work in the Tai differs because Tolliver's just wants to have data frames to be column stores with equally sized columns not objects that they pretend they can also behave like a rail frame no they have to be data frames in that sense yeah so this is really quite kind of the question like where does the data frame start or where does it end right what is it sort of how is it represented or how does it behave are two different things you can't mess with this and this is kind of this works then in the tidy verse so the in how do our individual geometries then stored well this is a slide that I created to sort of point out the seven simple things we can have we can have the point so a point is really sort of stored as a vector yeah so the thing that I did I showed you here the SD point so I know I'm going to construct a point and I need a two chord and it's like a co2 and it's a point with coordinates 0 2 but I can also have like I can have like three coordinates and then it says point Z or I can have even four coordinates and I have a point Z M so coordinates in simple feature framework really sort of most cases you will run into a two-dimensional have x and y but they can have a third and the third can either be Z or can be an M and it can have four and then the third is the Z in the fourth is the end I can also make an point M but I need to look up how that works then is X Y M like this X Y and Y a point M and now you wonder like what the hell is going on well it is it is you know to some extent convenient you be able to do Z because geometries are right our world of geometrical world is we think of it as three-dimensional yeah so it's not such a stupid idea to be able to represent points in space as three-dimensional coordinates right T is put aside sort of real world your physics and so on yeah that's good so that's the Z the M is a little bit more difficult story it's called measure in the standard and the standard is not very clear about what you should put into it but it's something else yeah so use the M for something else and that's a nice invitation to do things and you could you know you could think of properties of coordinates like you know what is the measurement error or so right if you take a GPS measurement you get XYZ coordinates and you get all kind of information about sort of errors in horizontal and vertical you could try to squeeze that back to a single number and sort of say measurement error you could also think of putting time in there right so it's a number we can encode time as numbers and we lose some of the reference but we can go back and so Anita grouse are one of the famous QGIS developers does that currently to represent trajectories and line strings so she has coordinates X Y and it puts timestamps into the M and you get a trajectory of course the challenge there is that if the trajectory itself intersects which they often do right we've always run back and forth to the coffee and so we'll consider yourself intersect with ourselves with our own coordinates then then the thing is not valid because there is not simple because this simple looks just at X&Y and doesn't look at that a third one might be time or so that is trajectory so so you can you can store things like that but then using you know using the machinery that you would like to use will be will be challenging in the in the next step so it's not like like trivial so this is what what the potential has XYZ coordinates in most of the cases we look at x and y because we do to your graphical data but there's room for is that G and M so a point is just a vector as we its printed here but it's really sort of it's it's really sort of what is this thing it's just it's just a numerical vector of length three with these with these numbers yeah it's nothing special then multiplying align strings are sets of points right both are sets of points and they're stored as matrix where we have the coordinates in rows and and variables in the in effect in the columns yeah so that is the simplest you you can get and then it's a two column matrix for x and y and three column for each Y Z and so on and then we get other things like multi-line strings so sets of blind strings and they're basically a set of line string so we make a list of line string matrices yeah it's a list of matrices for polygons we have a similar thing polygons are represented by a ring an exterior ring with zero or more holes in them yes so that might be 0 holes there might be more so any sort of ring that follows the first ring is considered as a whole and that again is a list of is a list of matrices right and then we have not to be gone multi polygons which is list of polygons yeah so we get sets of sets of sets of sets of sets and geometry collections are sort of a list of type geometries so whenever we have a set whenever we have a simple set of coordinates we put them in a matrix which is the most natural thing for for our to do and if we have sets of dos we put them in a list because they can be of all kind of sizes right a hole is typically smaller than the exterior ring and so on so so whenever there is a set of irregular things then it becomes a list whenever there's a set of points it's it's it's a matrix whenever there is a single point it's a it's a vector so it's really sort of takes the simplest form that I could think of doing this of doing this thing in R and this is so how things are stored so you can really that means that you really can get to the individual coordinates when you have an SF object in are you just need to accept the logic of nested lists yeah because at some stage you just have nested lists so if sort of if we if I have a single exterior ring in no holes it's still a list of length one with the exterior ring that is how these things are stored and then of course they're like your printed in in in this way like the way you see well-known text because you can print them as are objects but that doesn't really help much yeah so then if we if we use this package as F then we get this message I'm linking to Gio's library of this first indeed owl library if that first in approach for of dead version and so on and this gives a little bit sort of the current dependency tree of things around SF there are now something like 70 packages so the order are packed is using SF in one way or the other and SF uses a couple of other packages for instance it can use tiny force packages to do sort of spatial tidy things that that will show a few of these in a couple of slides but it doesn't sort of depend on it heart that is why this is a dashed line so you can use as F without sort of Tidy where's your tidy verse is sort of it's like 80 packages or so so it's pretty heavy so you can use as F without without doing this this is loaded on demand there isn't connector to DB I so that is sort of a generic package that's what has connectors to all kind of relational databases and that is basically used to read and write to spatial databases directly which is useful another way of doing that is through the GDL library you can also read and write to spatial databases through the data library so GDL is a library that or good ole as Roger calls it and in order people do it's it's mostly for it's mostly for I also it reads frustum reads and writes raster and vector data does a couple of web service things connections to databases and there's a couple of geometrical things but for that it really uses the GS library and that is there is a sort of a complexity that I think the complexity is Mac OSX when you want to compile when you want to compile things get things going on crumbs and you certainly want to have a binary package distributed on crown that uses external libraries you need to do a little bit of work yeah because cron provides windows binary and make I was expire ease for you yeah but it has to compile them so on their compile farms so to speak all these libraries have to be installed yeah now TTL is pretty much the probably the most challenging sort of it's somebody nodding back in the back it's definitely the most challenging sort of external dependencies that chrome and has ever had and still has because it's a library of libraries right it just doesn't do all these things reading and writing formats by itself but it uses like an array of like 50 other packages to do stuff yeah so for XML it uses lip expert for cheap JPEG it uses open JPEG for beautif it uses lip TIFF for you know an endless list of libraries that detail really sort of abstracts um yeah but it doesn't work on its own it really needs all these libraries now if you sort of you know if you are lucky and you work on relatively lucky you work on l'énergie have all this stuff installed on your computer you can say compile this thing below then the whole thing works and okay then your lucky did the thing that what cron does is it says I'm not going to make I'm not going to make binary packages that make any assumptions about your computer yes so if cron would say okay I will compile this data or this are GDL or this sf package for you but it will only work if you install Jade ah yes then hell would break loose because how do you do that there are five six seven different ways of doing that on on Windows there 15 different ways of doing that the mecca was X and so on so what Quran says okay we're not we're just going to distribute binary packages that are self-contained the only assumptions that these packages make is that RS on your computer yes and that you have a proper operating system yes that it runs Windows or Mac OS X in some version yes so these are the only assumptions it's not going to make any assumption so that means that the whole sort of stuff of all these fifty or sixty or eighty live libraries that are needed just to sort of have T doll working are basically in this package yeah so that is why when you're in Windows or Mac io6 you want to install SF or RG doe that you have a download of like eighty or hundred megabytes or something like that right it's installing all these libraries now you don't see it and you don't notice that's relatively good but it's it's all there and it's happening in somebody else took care of making making this work now why was he's making that point well it's it's a it's a it's a complexity yeah so it's something that that we we worked out and we are in regular Virginia regular contact with with oh and with Simon and so the people who do the windows and in the and omega-6 compile farms and that needs regular attention and updating and maintenance and things look easier now we have wind lab these days but they are not sort of they are still not trivial so that was tthe doll is a complex thing it's library of libraries it's a single interface to I think on my computer Deedle has under twenty dependencies or so so you can do you can look at if we can do that so the shared library of Lotito has as these dependencies yes has hundred ten other libraries that it links to so whenever I link to GDL it links to these things so if I look at my SF installation SF yeah and then in the Lib directory we see SF to the s oh yeah and we can sort of look at [Music] we're dead links to and you get the same story so this is dynamic linking it's linking 270 in library so that is digital stuff and a debt of course is augmented with lip R and so the standard the standard think that any our package will will link to yeah so this is dynamically linked so it is it is all sort of load at the moment I yeah so if I update T dulled and anything you know should work again automatically but when you when you do bind when you do binary installs on Windows it's a very different story so there's the tears library that does did us a geometrical operation yeah I know why why I went to this long story so the thing is that you can also use GDL for all the geometrical operations but that needs gos to be linked to it and that is something that doesn't work for some reason with static linking on Mac OSX yet that is why we need a different interface to G tau from the one to gos yes this is basically because the way cron is organized we cannot distribute packages that use Geo's through G Dao although that would be easier from a sort of software engineering perspective because then you have a single interface now you have kind of two interfaces yeah so des does all the 2d geometrical operations flat earth yeah does nothing with latitude longitude it does everything with sort of Cartesian coordinates then there is proj for and it's somewhat similar story but Cheadle can statically linked to proj for so we do use in SF actually the GDL interface to projections yeah so that's why it works slightly different from our G dough which does not do that yes which directly links to prod for and that's how it is and then we have links to lip lwd on which is sort of an library that comes with post GIS was developing a post area so that has all the things that post eis has but that are not indeed out there the items or some of them are good for instance vertical geometry due distances on latitude longitude and these kind of things and then we link through the unit's package to the UD units library which is us with C library that has units of measurement database and that does all the units conversions for you yes so it knows how to go from meters to us feet this is something you don't want to program yourself somebody else has done that right and the thing is that is with all these blue with all the blue stuff is essentially these have been done and this is not something this is not an hour problem but this is sort of an OST o problem that all the spatial you know is sort of open source special projects quantum T is post E is in anything that is not Java so to speak because Java has this Java hurdle sort of Java can only speak Java but anything that does C or C++ uses these these libraries this at left three are spatial the Uni units it's not special but it's kind of meteorology Oceano coverage or something like that the climate it's the you car sort of community that developed the netcdf sort of duty units netcdf community so these are all libraries that are basically used and maintained the larger communities that that know their way with this stuff yeah so this is good because then using the same software of course has the same has the risk that you sort of use the same bugs that is of course the disadvantage it's less rich in terms of in ecosystem but having to reprogram all this stuff is kind of an absolute sort of most stupid things you could you could think of doing yes so really what what we do is sort of writing interfaces rather than rather than do new stuff yeah but that is that still is fun so what are the features here is and now I have to hurry so if we extend data frames or or tables with geometry list columns there is sort of fast C++ to to and from our conversion there is spatial indexes that we use in eight years libraries that are on the fly yes this is a very cheap solution but seems to work pretty well that is something that was not present or relatively underdeveloped in argyus that is now you available in in SF so that when you have larger datasets then special indexes are your sort of your only way out of getting things done really it has a SF has a relatively small API compared to you the functions that are available in sp arterial and and argyus we see that there are much less functions in them I'm very hard to convince by sort of people suggesting account we have this function to do that and so on and and so I'm very sort of conservative in extending this the set of functions or methods available in this F and it has worked well sort of you know I have the experience of doing it not doing it the wrong way oh this is a nice function oh this is a nice function and then you can end up with huge collections of things that they are they have terrible names ready function starts with St underscore this is how post he is also does it so if we have like is simple then you say st underscore and it is sort of it is convenient because you can sort of on your on your on your our prom chicken you can say like you can say st underscore tap tap and then you get all the st functions and you can see what I do when if you work with our studio it's even much more crazy all the possibilities you have on the on the command line really but as you see I'm not so SF in the tidy first few slides so we can work with data frames or with tables you can always like do things you're on you can just work with geometry list columns but not with SF object you can add create geometry columns and put them in your own data frames and so on and then you you just you study verse and and everything should work in principle there's just that if you create SF objects that are kind of objects that are aware that one of the columns is a is a geometry list call them and and and it may be useful for certain things there are a couple of methods that are sort of dedicated or adopted filter arranged this thing group by and group mutate and select they all have sticky geometries meaning that that sort of whatever you put in the geometry will still be part of your output and there's an SD join that joins tables based on spatial predicates yeah so joining tables like merging tables merge in base R or join left join right and so on and in deployer sort of merge records based on attributes but we can also merge them based on spatial predicates for instance two things overlap or cover or intersect or something like that right so you can any throw any predicate in that and SD join will join you based on that predicate and then for instance summarize unions also the geometry by the grouping so it doesn't just compute group summaries on attributes but also by default unions the geometries like here so here we see little sort of printout of something where we do select from from Ty differs so we select only a couple of records or a couple of variables and we sort of controlled the significant number of significant digits of coordinates that are being printed here these kind of things yeah this is this is doing here you see basically we do an transform so we transform the coordinates to North Carolina state plane with unit US feet and that that is some kind of projection that you see here and you see also that they're still naughty polygons but you also see that the unit is now sort of related that we did we understand that these coordinates are now measured in u.s. survey foot which is the approach for way I think that you're the unit's way of of doing that here a couple of things that excited a lot of people that is the the Jim SF which now part of the TT plot - on crown where you can sort of plot the spatial data and with a nice thing that you see well I can see it but it's hard to see on the screen is that sort of the lines the white lines the TT plot are famous for are no longer straight but they are curved yeah I transform this again to you a survey feet but we don't see you with a survey feed we see that it feels longer to the longitudinal attitude and we see graticule zhan a background so they are no longer sort of straight lines but they are kind of reference graticule x' which is which nicely works and it also does automatic things if you sort of you want to plot something on top of that that has a different projection then it will auto project that to the projection of the first argument and so on so it goes in that sense it goes much further than what SF what is F it does yeah we can also put like you know use facet plots so the facet plots are kind of the composite plots whether you have shared X's here we have a shared x-axis and use that for spatial for collections of spatial maps that have different relate to different properties and this relates to two columns but of course to use this we have to first do the tidy data thing which means that instead of having these two columns that the disease cases the 74-79 as two columns we have to stack them yes that means that we have a single column that is now the disease cases and another column that says this is 74 this is 79 and another column that has the geometries twice yes because that is the tidy data paradigm that everything is in a long table that we don't have sort of a single variable distributed over several columns so we have to do that first and that is done by the tidy our package function god er yes God earth goddess things that is like what stacked us in in base R and then if you got our things then you can use the facet wrap and say this variable is going to be the facet variable right so people who do this daily find this very intuitive I can just you know I just do did it is once but it's cool that this is possible it's mean extremely cool it's this much cooler because this is map view right and there will be a lot of map view but this is kind of the nice thing is that this is the same but it's interactive and it has this background and you can sort of click on you know click on elements and you get sort of the whole sort of the whole set of attributes in this pop-up table so this is map view which is a layer on top of the are leaflet package which is a layer on top of the leaflets JavaScript libraries and so on so map view kind of can can do these kind of things you will see much more of that quantities as I said is a is so so we're working a lot with with data that have essentially measurement units so this is a little bit excursion into debt we can do for instance this polygon and and find out you know find out it create as polygons you can see this polygons are for points that have you know that are that are in Cartesian coordinates on the line but there are four points that serve that circle around the pole right so these are four points it's three points but it's a closed ring that that circle the North Pole and we can sort of find out what the area of this of this thing is when we say it's in projection language belong at you letting you there there with its seen as basically a circle around a pole and we see that the area in it is is this amount of square kilometers if we would say this is the equation circle in are also essentially Patkar a projection so let's get longitude our x and y and it is flat then we get an area which is of course 0 yeah you can you can see that these points plot us they are simply they're on the line right so it doesn't form an area because it includes the poles so we get the one thing to do is that we get measurement units when we compute quantities we can convert to other measurement units to make things convenient and also that depending on on sort of the way we look at data we get different outcomes yeah if I set my my current reference system to missing it will also assume it is it is Cartesian yeah their XY coordinates in some unknown space and you also don't get a measurement unit because it doesn't it doesn't even know what you know what a unit in distance what it means and we can also do the distance between these between the polygons in this point and we get a matrix back which is a units matrix distances in units in kilometers yeah so this is for you know this is for some sometimes this is sort of confusing that your numbers are not simply numbers anymore but in other cases it is sort of pretty helpful that in a sense that whenever you whenever you think that you that your values are in kilometers but they are in meters and you know that might be pretty disasters and and whenever this information is available you can essentially work with it now the second thing is that simple features are you know are nice but they're just do points lines and polygons and so the question is like where at the roster data so any story right now about you know any book that would be published like these days about spatial data in R would have like two stories like there is roster packets that does rest early and it does it the same way essentially that they did at ten years ago and then there are these you know features where a lot of people nowadays advise to use simple features for and they didn't sort of talk very well so this is sort of sort of work in progress to get you know to get a roster data framework that that kind of works with well with simple features and let's see where that when it gets so first of all raster data don't fit easily in the simple feature framework you know you do you always wonder like it's a raster pixel is at a point or is it a polygon how should I represent that or some kind of convolution over an area which it might be not rectangular because it's you know because it is some kind of a sensor or so we have we have we see raster data come up with continuous phenomena although you could represent things like like land use or soil type or so but these are always these are also often represented by by polygons but for continuous phenomena it's more natural to do like if you have air quality or temperature or elevation these kind of things it's it's Russell representation is more natural one because it's continuous and the the idea of a thing is not so not so clear there right so if you have elevation thing could be a mountain but what is you know where does the mountain start where does it end and it is very this is where it sort of geographic information science things to to think about all these kind of problems which are interesting but do more from morphometry person would say well here's the DM done yet story over we're not going to discuss where the mountain starts orient you just you have the DM go ahead and do whatever you want with it it's a raster a thing it's a pixel thing and so on you could you could or just a raster contain a thing you have you think about object delineation and so on data cubes it's um it's kind of a more general concept that kind of can include rosters but doesn't have to so data cubes are essentially kind of arrays so things data that are distributed along dimensions and in raster data have an X and a Y dimension I have basically pixels aligned with X&Y right but we can also have like multiple layers like time respect to bands this is what roster does in the raster package dozen raster stacks but then we can also have time end bands like for D data this is what was - does not do or we can have five D data where we have time time sequences of different spectral bands for different sensors like you know like sentinel-2a Sentinel to be Landsat eight and so on we can also have like time series for points you have we have air quality data hourly data four points and that also creates a data cube because we have the same kind of time discretization from multiple sensors yeah that's what it creates a create some kind of a matrix which is a special case of a cube data cubes are fashionable but not well defined watch out for opportunistic definitions I said there's a lot of you know politics people wanting to sort of claim the the the the topic of data cubes you know you find sessions about them and conference right now it's a pretty generic concept it's not new where this sort of goes back to the old lab online analysis and processing thoughts of data analysis and other examples are for instance health data where you have number of healthy and ill persons by region by year and by age class yeah so region is a sequence year is a sequence eight class is a sequence we have like a three dimensional cube a three dimensional array and then you have two numbers you had ill and the healthy person so it's forty air a these kind of things how will this work in our well we have to raster packages which is extremely powerful it works with pipes what tidy people like and so on simple features don't scale very well for us through a material data although I had a recent had a recent sort of meeting with with Mike summer as you can see here this is Mike's Mike summer was not mentioned by Roger in his talk but this is one of the art spatial person very active also very active from Twitter and so he does he has sort of his own sort of package verse sort of getup organizations with maybe hundreds of packages and a lot of these things too are very useful and and contain an enormous amount of good thoughts this is so and we discussed a little bit about sort of where raster data is should be going and actually in the simple feature framework we might want to sort of come up with a class that represents raster data in a relatively simple way right now the things things that I'm developing are in the stars' package star stands for space time tidy arrays which is when our consortium funded project and does raster and vector data cubes so array data with reference dimensions so our arrays will not have like information like what how does my dimension relate to X or Y coordinates or latitude and longitude or to time and these kind of things you basically keep track of those things so you can do more powerful analysis that deals with 40-plus raster data but also with yeah so that okay yeah so this so the the challenges that we want to go on with this stars project is to deal with higher dimensional data so more than three dimensional rosters more than raster stack so to speak and also with data that sort of don't fit on local disks yeah this is another limitation Ruster basically says well I don't care about your memory but I but I assume that all my data is on the hard drive which is also a limitation and if you think nowadays of working with sentinel to data then that might not be sort of a realistic proposition unless you are very patient with downloading and organizing other non raster data like origin destination matrices can be represented easily by this and this is a work in progress to be finished hopefully by the end of this year here's a little example where a data set that Roger once selected a Landsat data set is it's plotted so you see different slightly different plot methods then the raster package has so it's a it's a it's a stack of six rosters where they have a joint joint legends where the legend classes are basically quantiles of the of the joint set here is in Sentinel 5 data set so this was it is this was like July when the first Sentinel 5 P data came down is a one of the Sentinel satellites that measures air quality and I gave it a go and sort of looked at and it's it's raster data but the thing is if you look at this roster so they're sort of you know it is like hundred by hundred great but it's not like a regular roster right it's a curvilinear roster so this is a supplied sort of taking a funny path of the year and then you get something you get something like this so this is sort of no-go when it comes to really reading things in qgis or so Deedle can read it but there is no let's say the spatial referencing is for every pixel you get a levitt you'd and longitude value and so that that really sort of leaves you to how to deal well I need to talk to the grass people how they deal with this but this is sort of a no-go for for Ruster and for cue quantum GIS so you can't say read this netcdf file you have to sort of go into this this is what I did and this is sort of the terrible code how it looks like so I sort of looked at what kind of this netcdf first of all the netcdf file has you know if the file name is like two or three lines or so so I cut out a file name so so you could still look at it and here it falls off the screen but I reading the latitude great which is a raster for every pixel has a latitude value and then the longitude grid which has launched it for every grid cell and then the nitrogen and our two nitrogen dioxide values that is what the thing that I'm measuring my sensor data here you can see sort of the latitude values and you can see it doesn't have any referencing because this is the referencing yeah these are the latitude coordinates for every individual grid cell then I'm setting the n/a mask because the nitrogen values didn't do that by themselves must be a mistake because netcdf can do that and then I hacked some some conversion to as F function that takes these two matrices and makes that into a points object really so I get points with x and y let it longitude coordinates out of these matrices and every nitrogen value SD as the as the attribute and this is this is a point data set that I'm now plotting which is okay you don't you wouldn't notice you can notice from the round corners that these are points they're round symbols but I didn't tell anyone so conclusions is that we have now on SF package already for a while and it's getting more and more robust it provides tiny spatial analysis also makes us nicely with with tidy verse stuff for people who enjoy working that way and works well for factor data it has six package for yet for further reading there's work in progress on tidy roster data space time array data that that happens in the Stars project a lot of things go to the our spatial community and to the our consortium funding the SF and stars packages this funding it's not it's not that it's so much money but it's really sort of makes me commit to stuff yeah so it's really sort of makes me more productive and and then not do other things of course that is the bad thing but it is it's fun to do actually for further reading there is now since one or two weeks ago there is this there is this our journal paper out called simple features so this is now the canonical reference of the simple features for our package that sort of also summarizes a lot of the stuff that that in sort of expands on some of the stuff and and sort of summarizes this talk essentially and gives you the references to all further references to all the standard documents and other packages and things and and thoughts how do I get back okay right and there is the our spatial dork website where there is also blocks on developments on stars and so on are always plotted there and there is other developments published there if you have blocks that you would like to have so our spatial that arc gets a lot of traffic if you wave so we I'm very much open for guest blocks if you have stuff that you think fits in there and want to see their published then please contact me or ask or sent a pull request and so on that works worked quite nicely contributions are very welcome there so thanks for your attention so we have time for one or two questions yes oh yeah that's a good question so I think that I think that it is not there is sort of some things work I think that there was a roster package update where a couple of the functions now accept a safe object as input it will simply convert it to SP objects at the moment it gets them but roster is a package that has sort of evolved from doing roster analysis into a package doing pretty much everything so it even dropped roster analysis at some stage from the title it's a spatial modelling and indeed it did sort of do all the things that are Tia's wouldn't do like combining geometrical operations with attribute things that things stuff that now by the way SF does do I didn't talk much about it but at that moment I don't think that if you feed it with SF objects that it will return you as if objects in the cases where it generates point I'm polygon data so I think it will always like return SP objects I'm facing the same problem so it's say what Roger and I do is sort of first write basic packages and then modify analysis packages so I'm looking at now at Gstaad for instance how will that work with with SAF and with stars and still work with SP things right how can it work with with data into into different formats and I did that once because originally it worked with data frames and then it worked with data frames and with SP objects but what would do either way right so so you get you get ugly code but things that are useful for people who have you know ten-year-old scripts that you won't still want to serve yeah so you know we're facing some choices there the what what you know you can look at what our studio does is just change everything completely every couple of years and get every everything mad but then but then there is you know a lot of energy and and help and so on and compassion yeah so yeah so roster is not the it's challenging itself its massive package and we met up we had an also during the our studio conference we had a meeting with HN Racine and and Robert hymens to also discuss stars and we're roster is going and so on and it's not entirely clear Robert is thinking of of doing a new package essentially and I don't know exactly where it is or whether it's going or whether it will be it will be good of course but that's all I know yeah good yeah Robin thank you know we have try this one I don't know if it works just building on that previous question and being more specific I remember seeing a message on the our CEO list from Robert saying yes I'm going to continue to maintain roster but I haven't seen many updates there nobody but then the question is what is maintenance yeah this would be new development really rather than maintenance yes that's my question do you think that it's most likely that there'll be a replacement package for rust or it will just disperse into different things in the future yeah well did this that is hard to tell yeah that is so so as I said roster is Russ tourism is a massive package right that there tries to do everything and I would never try to to do to write something like that because I know how much work it is yeah there is ten person years or so so I'm not you know I'm not going to do this and I don't see anyone else doing that so so Robert may may manage and maybe maybe he doesn't and we we now see that a couple of things a couple of things that a lot of people do rasterizing polygons right so going from polygon to raster or from raster to polygons these two things that you can do with raster but then there are all there are three other packages that have you know better performance or that can do this faster or that better or that more clear or something like that so these are kind of the the the sort of the low-level stuff that that many people do all the time and where it pays off other packages do raster analysis faster because they say well my rosters don't have missing values so you and you can do everything twice as fast right this is the same thing without rap that can also do things twice as fast when you know that a vector doesn't contain missing values you can skip the loop right you know it doesn't and so yeah I think I think we will see some diversification in the next couple of years and maybe some kind of you know convergence after that let's see

Channel: Tomislav Hengl (OpenGeoHub Foundation)

Views: 2,107

Rating: 4.9166665 out of 5

Keywords: OpenGeoHub, R spatial, Open GeoData

Id: yhpkx_xO-LE

Length: 86min 4sec (5164 seconds)

Published: Mon Sep 17 2018