Review: iPhone LiDAR scanner + Photogrammetry | 3D Forensics CSI | CLICK 3D EP 13

hey everybody it's eugene lisho here and welcome  to click 3d this is the program where we talk   about photogrammetry and how you can use your  digital camera and some software to make some   really amazing 3d models today i thought we'd try  something a little bit different and you can see   here this is the iphone this is the new iphone  12 which has lidar capability so i'm going to be   testing the phone to see what kind of data we can  get out of the lidar now you may be wondering well   how does this relate to photogrammetry well in  truth what i'm going to be doing is testing an app   that's called everypoint and the everypoint app  does an interesting thing it actually leverages   the lidar but it fuses it with photogrammetry  and so when the lidar sort of drops off you can   depend on more of the photogrammetry data and when  the photogrammetry data has some problems you can   rely on some of the lidar data so there's been a  lot of noise online with respect to you know the   the the iphone 12 lidar and you know really simply  creating some uh meshed data and stuff like that   and to be honest with you i haven't been all that  impressed with the data it looks pretty good but   in testing that i've been doing like the meshes  come out really lumpy they're not highly accurate   um you know they might be okay for doing some  augmented reality and just kind of playing around   and getting some uh you know cute uh models and  stuff like that but what i'm interested in here   today is accuracy how accurate is it really so  we're going to be doing three things the first one   is we're going to be scanning a vehicle and we're  just going to be using the raw data that comes out   of the iphone so basically you know scanning and  then taking a look at what the point cloud data   looks like okay how accurate it is you know where  it maybe breaks down or where it performs really   well then what we're going to be doing is we're  going to be trying it again with the every point   app and that's the one that will take the  lidar data and the photogrammetry data   and then fuse it together finally what i'm going  to be doing is i'm going to scan the car with a   terrestrial laser scanner it's the ferro s350  it's the main instrument that i use for my work   it's dependable reliable and i know it's super  accurate so you know in the ranges that we're   going to be working in you know it's probably  on the order of a couple of millimeters or so   so i'm going to take the data at the end and do a  full 3d comparison so i'm not just interested in a   few points like here and there and just comparing  measurements i'm really interested in the overall   point cloud data to see how it performs okay so  let's get started i'm going to get outside now   i actually did record some of this the other day  but i'm gonna be doing the lidar part and uh just   showing you pretty much what i'm gonna be doing  uh moving around the car so let's get started okay so we're gonna do the scanning here with the  the everpoint app and what i'm going to do is i'm   just going to fire that up and i'm going to choose  just basic lidar okay so there's nothing really   too crazy coming out of here  and i'm going to choose a dense   sort of medium density for the points and  i'm going to make three loops around this car   so let me start from there i'll start from this  actually i'm going to start from the middle of   the other side i'm going to go up high and go  around then i'm going to go around the middle   and then i'm going to go around the bottom  so let's give that a go all right here we   go so i'm going to start the scan up high  and i just want to make sure that i've got   the top of the vehicle here i want to make  sure i get those points so i got to get up high   and one thing i will say about the new iphone is  the tracking or you know the uh the accelerometers   or the gyros or the imus whatever they got in here  whatever magic they got in here it really tracks   fairly well so that part there is definitely in  our favor and you'll see i'm uh well i don't know   i'm a couple meters away from the vehicle as  i go around so this is my second loop i'm kind   of focusing on the main body of the car and when i  wrap around the other end i'm going to go down and   sort of focus on the ground a little bit more and  i want to make sure to get these parking lines in   there at the bottom because they're kind of my oh  they're kind of my guide over here so i'm going   to make sure that i get these in at the bottom  here get down at the bottom like that okay and   almost done here okay so getting down low and  that's about it so we'll have a look at how that   works out and what i'll do is i'll show you what  i did uh with the every point app using both the   lidar and fusion of uh lidar and photogrammetry  and then we'll come back to that and i actually   recorded that the other day but i'll pop it  on here and you'll have a look for yourself   hi everybody it's eugene lisho and here today  i've got a couple of instruments beside me   uh one is an iphone the new iphone 12 with lidar  on it and i also have a ferrofocus s350 scanner   today what i'm going to be doing is doing a check  of the accuracy of the lidar on the iphone and   actually it's not really the lidar it's actually  an app called every point and i want to check to   see what kind of accuracy we can get out of this  so to be honest with you the lighter that's coming   out of the iphone is okay but i'm not really all  that impressed with it so what's interesting about   the everypoint app is that they're fusing two  technologies so they're using photogrammetry   and they're using lidar and they're putting  it together so when the lidar drops off the   photogrammetry will provide you with some better  data and vice versa so that's really interesting   on the ferroscanner side here this  is a terrestrial laser scanner   within the ranges that we're dealing with we're  going to have like a one to two millimeter range   and so this is gonna be our let's call it our  ground truth and we're gonna compare this to   this and we'll probably do that inside of cloud  compare so what i'm going to do now is i'm going   to set up to document this vehicle behind me  and i'm going to do that first with the iphone   and then next i'm going to go with the laser  scanner so let's get to it okay so i've got   the phone here there really isn't much to it i'll  show the screen uh separately on the view here but   basically i'm just going to choose video plus plus  lidar so i'm going to do that and then i've got to   name it so i'm just going to give it a quick name  i'm going to call this my car like that and i'm   going to go ahead do that a window pops up it has  like a depth map on it and so what i'm gonna do   here is i'm actually gonna go around the vehicle  maybe three times so i'm gonna start up high   do like a mid and then go down and do a circle at  the very bottom so i'm going to tilt this just a   little bit just so i get down and like that and  i'm going to start scanning in a second here so   i'm going to go up high i'm going to start okay  and it's flashing so it looks like it's starting   to scan so i'm just going to move around like  this and start going around do my first circle   and of course i can't really see the phone  so i'm just kind of backing off a bit   just to make sure i get everything that i need  and i guess i should have also mentioned that   i'm going to use the parking lines on the ground  to create a kind of set of controls so i'm going   to make sure i get that there and i'm going  to turn back and get this part of the car   so now i'm going to go down to about eye level  and i'm going to start moving around again   like this uh it's kind of shiny and reflective  and many of you might know that cars are not the   best for this sort of thing because they give  a lot of problems with reflections and such   so i'm just going to go around this way and  i'm almost completed this next circle and the   next thing i'm going to do is get down a little  bit lower and do the bottom part of the vehicle   alrighty so we're back on this side now i'm going  to start dropping down and i'm just going to make   sure that i get just nice and close to the  bottom and part of the ground get the tires   okay around the front and i'm going  to do the same thing on the other side   and then once i get back we should be done and  there seems to be enough texture on the ground   i should be okay but i'm going to make sure  i get the back end down a little bit lower   okay and then i'm also going to make sure  that i pick up the lines here on the ground   and that's it i'm going to stop this right  here okay so i'm good so i'm going to go done   and that's it that's pretty much  it for documenting with this   now um in order for me to finish and get an actual  product or some kind of a point cloud out of this   it has to be uploaded and processed and then  i'll get the the point cloud downloaded again   so but that's pretty much it  the documentation part is done   it's pretty simple three circles around the  vehicle now let's switch over to the laser scanner okay so this is the ferro laser scanner and on  this one i have it set up so that's it's at a 1   8 resolution at three times quality and basically  what that means is at 1 8 resolution i get about   12 millimeters of point spacing at 10 meters so  if i'm at one meter away from the car then i'm   gonna be at about 1.2 millimeters so i should have  a really really dense point cloud and we're going   to go around and do the same type of thing from  a number of different positions this is going   to take us a little bit longer because it's going  to take us over three minutes for every scan and   that's for the laser scan and for the photographs  so let's get started i'll start from this end over   here and we'll plant that here level it off a bit  i'm going to move this up just a tiny bit higher   and that's it let's wait and see  what happens and we'll go from there okay so we're done with the laser scanning right  now so what we're going to do is we're going to   take all this data we're going to process it  and then we're going to see how they compare   so let's go over to the computer okay so what i thought i would do here is just  look at some of the scans that we did for the car   and i actually tried this three times and i'm  going to show you all three on the phone here   it's not that promising right off the bat so  in looking at this car you can see there's   all kinds of artifacts here so especially on  the side it's there must be something wrong with   going around three times it doesn't actually  register these you can see there's like   two other doors there on the side so there is  something obviously going wrong here even on the   roof here on the top and i would expect to be  some noise but this is just uh terrible here so   i actually tried this again i went back and let's  look at the second one and see what we got here   so this is another trial and you can see that  i've got a lot of noise up on the side and again   the the top of the roof here is giving me some  problems but when i go back to the side again   i can see that there's um sort of the top part of  the door has sort of come undone there and it just   doesn't look crisp there's all kinds of noise and  i can see there was some mismatches on the ground   as well so i'm just not all that  pleased with this i can see here   that it just looks really really noisy a lot of  noise off the side and then even this other one   that i just did kind of quickly going around just  to see if there's something strange going on but   again if you look at the door here there's  a clearly a problem here and on the roof   again you know a lot of a lot of these noisy  points that are coming off on that area so   this is not something that i'm all that eager to  test and i can see that there's some problems here   so i don't know i may just give up on this  because this is not what i was expecting i   thought it was going to be a little bit better  than this but maybe what i'll do is i'll bring   in one of the point clouds in cloud compare  just so people can see and we'll go from there   all right so i brought in uh some of the data  from just the lidar points only uh that comes   straight out of the iphone and you'll see that  it's not all that attractive and this is the   one of the issues that i had where i had these  two doors so for every loop that i went around   for whatever reason it didn't clear out this  part or didn't figure out that it was registered   on this part of the vehicle so you can see  there's quite a bit of noise you know it's   not really crisp where the license plate is  and up top here a ton of noise off the edge   so i'm not all that impressed with it actually  i just noticed here too on the back side of   the vehicle here you can see there's actually two  sides here so it's kind of overlapping at the back   so for whatever reason uh the uh creating loops  i thought that was actually gonna help give it   a little bit more to work with and maybe do  some kind of a registration where it closes   out the loop but that's clearly not the case so  in this regard this is not a great model and let   me switch to the other one so i'm going to shut  this one off this is the second one that we did   the third one had the same problems but you  can see you know i've got a lot of noise here   a lot of noisy points off the top and a lot  of points off the side so there's clearly some   issues this was in the shade just so you know  it wasn't in direct sunlight it was um there   was a bit of sun in some cases but it wasn't in  this particular area it was being blocked by the   building so um this is what we've got maybe what  i'll do is just to kind of continue with the test   is i'll crop this vehicle out and i will clean it  up a bit just uh give it a little bit of help here   and then we'll see how it compares with the uh  the laser scanner data and we'll go from there   so let's look at the data here and i'll show  you what i'm doing here and sort of explain it   so the first thing that i want to do is just  look at the general models and sort of say   you know how well they look in terms of  cleanliness and crispness and also in terms   of scale so what you're looking at on the screen  right now is the ferro laser scanner data and   you can see that it looks quite crisp it's quite  clean the ground is uh very very clean and crisp   and so this is why we're using uh this particular  model as the ground truth now the the top here i   could have gotten a little bit more data by  getting a little bit higher with the scanner   but that's okay i think for what we're after today  it should be absolutely fine and as you go to the   other side same thing nice and crisp clean data  you can make out a lot of the little details which   is great now on the parking lines that were on the  bottom here i took a quick measurement and it's   about 6.087 so let's say 6.09 it's getting close  to you know 6.09 or so and that's sort of from   this corner to this corner on the on the ground  there between the lines so this particular model   looks really really great let's look at the  just the lidar data from the apple iphone   okay so this is the apple iphone lidar data  and i've done a little bit of cleanup on this   not too too much but i actually did more cleanup  on another model and i'll explain that a bit later   but you can see right off the bat it's very fuzzy  there's a lot of noise not as crisp on the details   but you know generally okay and when i take a  measurement between these lines here uh where   the parking lines were i get 6.04 so that's about  you know almost five centimeters uh different so   there's obviously a scaling issue here or some  kind of an accuracy issue just right off the bat   so that's okay you know for a general model uh  this is the starting point and i'm just looking   at this briefly what i'm going to do after is  actually rescale the model or basically register   all the models together to the point cloud and  then that will adjust the scale to get them closer   so let's move on to the every point data and  let's see what that looks like okay so this is   the every point data and right off the bat you'll  notice that i'm getting a lot more around the   vehicle and on the building and i thought that's  interesting but this is now where we are combining   the lidar data from the phone with photogrammetry  so it's capturing video and it's processing those   frames to help reconstruct and recreate the model  the one thing you'll notice right off the bat   when you look at this model is how crisp it is the  other thing that i'll note here is that the colors   appear to be more true even when compared to the  laser scanner and that just has to do with you   know the video or the sensor that's in there the  data on the side of the vehicle looks really good   not a lot of noise there's noise down here in  the bottom and i found that in scanning vehicles   there is a bit down at the bottom here typically  with the everypoint app but um you know easily   removable there but i really like how the sides  look on the roof here that was lifting up the   phone to get up here but you'll notice that i get  some pretty good data and you know from the lidar   and it seems uh full or or relatively full and  i really wasn't trying all that hard and you saw   how quickly i went around this car and i did  three loops now unlike the lidar data you'll   see that i don't get any doubling up of the doors  you'll see that back here it looks very very crisp   so that's good too now on the scale on the  ground here you'll see that i have 6.13 so   on our laser scanner data we had about 6.09 so  we're about four centimeters higher so on the   iphone app lidar just the lighter alone we  were about four or five centimeters lower   and here we're about four or five centimeters  higher so uh neither appears to be accurate um   highly accurate anyway uh right out of the gate  in terms of just the some of the scaling here   so there's a scaling issue and some of that  needs to be corrected for which we can do   in cloud compare so let me explain  what i'm going to be doing from here   so that it's clear and i'm going to take each of  the three models i should say the two models i'm   going to be taking the apple lidar only data  and i'm going to be taking the every point   data which is the fusion of the lidar data and  photogrammetry and i am going to scale those well   first of all i'm going to isolate some of this  data clean it up a bit more and then i'm going   to scale both of them to the laser scanner data  so they'll be sitting right on top of one another   and when you register i can also adjust the  scale so basically to make it a fair test   what i'm going to do is i'm going to try and scale  each of the two models as closely as possible to   the feral laser scanner data which is our ground  truth once i do that then what i'm going to do   is i'm going to go in and i'm going to do a  cloud to cloud comparison so instead of just   looking at a couple of you know individual points  here what i'm going to be doing is looking at the   overall model and looking at you know millions  of points and seeing how they fit within one   another so let's try that out and when i come  back i'll just have that ready up on my screen   all right so i have aligned all the data and  made all the comparisons and so now it's time to   look at the results so what i have this is the  ferro laser scanner data that's in front of me   and what i'm going to do is i'm going to turn on  the every point data so you can see that's sitting   on top there and i'm going to turn on the lidar  data only from the iphone and you'll see that   that's sitting on top so they're basically  all on top of each other and i've tried to   you know scale them as best as i can and get them  aligned as best as i can so the comparison here   will in fact depend on how well you align these  so there can be some uh plus or minus differences   if somebody were to try this on their own  uh however um i think uh based on the type   of results that we're getting you'll get to see  the um the general idea here so let's start with   the let's start with the lidar only data so  let me shut off some of this let me go into   the lidar only and let me show you the scalar  field so what i'm going to be showing you here is   how well the points compared to the ferro laser  scanner data okay so when they're sit when the   two vehicles are sitting on top of each other it's  going to start with the faro data and it's going   to go out and start looking for the closest points  and then it's going to document that particular   distance so what you're looking at this particular  heat map blue is good wherever we get blue color   that means that the errors are low and as you  start moving to red that means that they are   going to be at a close to about five centimeters  or greater so in this particular model you can see   that i've got some red around the front and you  know it's getting uh in between on the yellows   on on the bottom so this side is you know blue  and yellow there's a little bit of a mix here   um but um you know this side's blue and then it  goes red so there was some there's obviously some   problems with some of the points that are  out here um but again um you know it's it's   okay for an overall model i think uh if you're  looking for several centimeters of accuracy um   but if you're looking for something that is much  lower like on the order of a couple of millimeters   then you know this is not going to be useful and  one of the things that i like to do here is i like   to look at the histogram for this so if we break  this out this is a distribution of all the points   up to about five centimeters and what we can do  here is we can put a little slider across here   and what i'm going to show you here is that when  you go to one centimeter the one centimeter mark   it's going to tell you that about 54 of the points  okay are below one centimeter which is pretty good   and then when you start getting to about  two centimeters you're about 78 percent   and then when you get to three centimeters you are  at about 92 percent okay and of course if we go to   five centimeters here like way out here we're  almost at 100 of the points okay so the the   cap here is about five centimeters but um again  what what i'm looking at here is how many of the   points are within one centimeter and we're down  around the 50 percent range okay let's look at the   every point data now and i'm going to turn that on  here and i'm going to shut off the ferro data and   i need to turn on the color ramp so just give me a  second here and turn on the scalar field okay here   we go here so what you'll see here is a similar  kind of thing you'll see a little bit of blue blue   we don't get as much red but we do get some red  up on the top here okay a little bit over here   and a little bit over there um on the sides  blue okay so you're getting a lot of blue   uh sort of around the edges of things and on  the ground for sure so that matches up very well   with the ferro laser scanner data but let's  have a look at the distributions now and see   what uh what they they look like and i've set  the scale the same so they're both set to cap   off at about five centimeters anything above  five centimeters i'm not going to worry about   that's way too much anyway so let's open up the  histogram here and let's look at how these compare   so right off the bat i can see that there's a  lot more points and if i go to the one centimeter   mark here okay somewhere around here or so i'm  getting close to 74 75 percent okay so before we   were down around 53 or so now we're at about 73  73 and a half percent that means that there are   a lot more points which are more accurate which is  good and now when you go to two centimeters here   okay we are getting close to 90 percent we're at  about 80 percent so what that tells me right off   the bat is uh we for us to get to 90 we had on  the other vehicle or on the lidar points only   we had to go up to at least three centimeters but  here if you go up to three centimeters you'll see   that we're already at 95 96 percent so what this  is telling me is that the majority of the points   uh by fusing the lidar with the photogrammetry  whatever magic they're doing there between the   two is definitely helping the overall accuracy so  that's really really good to know the other thing   as i said in just looking at the point cloud if i  go back to the color there's no denying that the   quality of the data here the crispness the colors  and everything else just looks much much better so   i think i'm going to leave it at that i'll let you  guys be the judge there but you know clearly there   seems to be a benefit with fusing two technologies  together with every point and um i want to thank   every point uh the people there for um they've  made everything accessible to me they've been very   honest about letting me talk about all the errors  and everything else so um good on them but i   really like the data that they're providing here i  think this could be extremely useful for you know   whatever if you're doing accident reconstruction  and that sort of thing and i'll show you one more   thing that i was able to do with the app before  we leave here so just to hang on while i switch   screens okay so this is the final thing that  i'm going to be showing you here but this got me   pretty excited because what i was able to do with  the everypoint app was i put it on a poll just   like you saw me in this particular video and i  walked around the parking lot near my office here   and this is the data set that i got with about you  know 10 minutes less than 10 minutes of just kind   of walking around maybe five minutes of walking  i just held it up on the pole walked around   and you can see here that i was able to get this  entire lot now i have done some tests on this   but i won't report anything just now but let  me just say that it looks quite promising   but as you can see getting information like  this from just a phone is super useful and   the fusion of photogrammetry and lidar i  think is a great idea it's a great concept   and a great technology so i'm going to leave  it at that and just make some closing remarks   well there you have it folks i found that pretty  interesting trying to see what the raw iphone data   looks like when it comes out just the phone as is  without any kind of you know special treatment or   anything like that and also the every point app  which fuses photogrammetry and lidar together   i was really enthused about what they have there  i think they need to do some work and i believe   uh from what i understand it's still  at the initial stages there's a lot of   work and development that can be done but from  what i'm looking at initially the accuracy that's   coming out of the everypoint app is much better  than you know maybe some of the other apps that   you're going to be using with the iphone so that  does it for click 3d i want to thank you all for   watching and there'll be another video coming  shortly thanks a lot and have a good one bye

Channel: 3D Forensics

Views: 49,986

Keywords: photogrammetry, 3d scanning, 3d scanner, reality capture, laser scanning, 3d point cloud data, 3d scan point cloud, land surveying, forensic evidence, 3d model, 3d modelling, Vr, virtual reality, forensic scientist, crime investigation, accident reconstruction, zephyr, photomodeler, 3dsmax, cloud compare, archaeology, Autodesk ReCap Pro, Agisoft Metashape, Meshroom, Pix4D, 3DF Zephyr, Regard3D, WebODM, Drone, aerial data, RealityCapture, LiDAR

Id: -uCM58WHA6U

Length: 29min 39sec (1779 seconds)

Published: Sat Mar 06 2021