Segmenting remote sensing imagery with the Segment Anything Model and box prompts

hello welcome to another segment geospatial tutorial in this video I'm going to show you how to segment limousine imagery with a box prompts first let me show you how to download the notebooks so if you go to the website same geo.gshap.org the link is also in the description and on the left side click Noob examples and then box prompts so this is a new feature that I just added today uh essentially allows you to segments remote sensing images with multiple input prompts makes it really easy to use you can provide any Vector data if if you already get some bounding boxes from some other machine learning models or from whatever data sources they you you can get and so you can use that to inform the segmentation so if you scroll down to the end I can quickly show you the results so the yellow bounding boxes are the one that you can input into the model so given an imagery and then you can provide the mounting box then we can utilize the segment anything model to segment objects so basically we find the object not just the bounding box now you really get the object within the Bounty box it's really useful and very easy to use previously I already um created the tutorial that shows you how to segment using points or text prompts so this is the Box prompt is another requested free so that I just Implement uh today so it took me a while to actually to um get it done and so in this video let me show you how to do that so upper right corner if you click the download icon then you can download this one to your computer once it's downloaded then you can open using um oops open using Jupiter lab so I'm going to just uh at jupiter.net and hit enter it's opened it in the browser if you don't have a GPU so this is recommended you have a GPU if you don't you can try it on Google collab so just right click open your new tab and it's open this one in Google call it and once it is Open click runtime change the runtime type you want to change this one to GPU CPU is pretty slow so if you're trying to GPU it should be much faster and then click save the first step is to control slash to uncommon this one and you just click run then just follow step by step you should be able to run the notebook just using the browser without having to install anything on computer the only downside is that you have to install the package every time and also the the map displaying the master is little bit buggy it's not as smooth as running locally so but you're welcome to try it out if you want and for the remaining of this tutorial I'm going to just show you how to use it locally and I assume that you already didn't want this I already installed the packages if you haven't you can just go to the installation and then just use a Mamba to install the package and then you should be able to use that so first step is to import the libraries and we're going to import live map and also same Geo we import two basically two functions the first one is the TMS to geotif allow us to download Time map service to a cloud optimize YouTube and the second one is the same zero the class that we can use to secondly imagery so first let's create a map and then center around somewhere latitude longitude wherever you like the zoom level and also the height then we also add a base map so take a look at this here um um this is the best map so behind the scene is the Google satellite base map and once you have that you can specify reasons so you can use the join tools to do a rectangle on the map then we can download it so if you don't draw the rectangle is going to use this by default or this money box so the latitude actually x-min x max y Max so basically just the lower left and upper right corner the coordinates with that then we can use the function to download images so TMS to geotif and the output this is where you want to save the images so we're going to save that as a file name give me stock TAF basically on the under the current walking directory and then the value box you are specifying where the reason you want to download the data and the zoom level and the source basically the the source of the base map so this is the Google satellite but you can try out some others and the last one is whether or not you want to overwrite so just run this one you see it's pretty quick so uh it's basically downloading the tiles and then Moses them back together as a single zerotif then on the left side here you should be to see this image image talk teeth so this is how you can download images of course this is just for educational purpose only uh if you're trying to download a large amount of data uh you probably divided the term of service so I always make sure that you have the license to use the Imaging but there are tons of other images online and they allows you to use 3D the reason I use this one because it's also at the global scale so you can try out any way you like but yeah don't try to download a lot of data okay so once you have the Imaging then we can just add the image on the map and this library's key is to hide the base map and then add the last day we downloaded so if you want this one again now you should see this image video we downloaded so this is the image stock teeth the Imaging you can upper right corner here click this one and then click here you can turn the layer on and off you can also change the opacity if you want the Google satellite basement is still there by just hidden so this is the map basically the images here opacity so let me turn it back on okay so you see it's pretty clear and what we want to do is to for example segment the trees from this image right so we're trying to do the manually DNA so it's going to take a while uh you can you're also welcome to use other deep learning models to do that but you need to have training data and then you need to build the model and then you do the segmentation into the meditation by using the second geospace so only just imagery having the mounting box and then you can do the segmentation without requiring any training data so next let's see how we can do it so you need to initialize the same to your class and this one allows you to specify the model type and also the checkpoint and whether or not you want to be automatic since we're going to provide a bounding box we don't want the automatic mode so this basically disabled it and just one this one it might take a couple seconds so this is going to initialize and this step might be uh take 10 20 minutes depends on internet speed if this is the first time you're running this one you don't have the checkpoint yet so the checkpoint itself is 2.4 gigabyte uh since I already downloaded on my computer and I've done it before it's uh already downloaded on my computer so I can quickly show you uh where it is located so it's under the home directory and then on the case uh from here I think it's torch and then Hub checkpoint so this is the checkpoint it has been downloaded 2.4 gigabyte if you don't have this on your computer it's going to take some time to download so just be patient once it's finished downloading then we can just set the images so this is uh the step that you need to do is to specify the image you want to do the segmentation so the this variable if you come back to this step here is so basically this imagery so we want the second main segment this image and just shift enter it's going to take some time so depends on your computer configuration this can take one to two minutes um just be patient because it's reading basically it's reading this checkpoint loading into the GPU and then so that we can do the segmentation so this is the uh most time consuming steps of the entire notebook just make sure that you wait if you have a very small GPU sometimes it might slow uh um RAM memory issue that means that your GPU you run up the GPU so you can either try a smaller imagery or you need a larger GPU or you can try Google call it or other cloud computing platforms that is a sufficient GPU so all the segmentation stuff computer vision you need to a decent GPU okay it's done so take a look here here we already finished this step and then the next step is to show the map again so we're just going to show you what I'm going to do here and there are different ways you can provide the bounding boxes you can either create a bounding boxes interactive on the map or you can just provide a vector uh bypass to a vector data so I'm going to show you two ways the first way is to create the value box uh boxes interactively and all we need to do click the join tools here and there are a lot of trees so if you want to extract a specific one for example I want to extract these three so you can just do a rectangle around this tree somewhere here right and you're welcome to extract more so you need to click again and maybe this one okay like this you can oops so this is the back of the iPad leaflet if it doesn't stop you just press Escape key on your keyboard then it should be done so here basically we just do two two rectangles and we want to extract the trees within this rectangle um all we need to do is draw the rectangle and then since we drew the rectangle so the rectangle is not none if you want to check it you can click a new call block and then just type m dot user allies because we we do a multiple rectangle so if you want to get more um both of them you need to use the ROI so this one exchange resistor Geo uh Json dictionary so you can see within here we have the coordinates and we so also have basically have two so if you do more you want to have more geometries within this dictionary and once we have this then we can um secondly Imaging so what we want to do is to call the same zero class right so if you go remember here we create the same Geo class and then we specify the imagery then you can call this function to critics so the predict except this same class already have the image encoder so you already have the imagery then for the predict you can press a shift tab on the keyboard to see the input parameters right so you can input the point coordinates uh you can also provide the label but in this case we're just going to use the bounding box you can use them together if you want to but for simplification I'm just going to provide a bounding box so the bounding box you will see here the bounding box is this one so basically the Bounty box are the two one that we just saw there is coming from the M.E user Ry right so if you draw something it's going to pass those two money boxes to this variable if you don't do anything it's just going to provide a list of two Bounty boxes again this is from the uh uh x mean y mean x max y Max so you're welcome if you know exactly location the coordinates you can provide them but sometimes it's pretty difficult so it's easier to just do a rectangle the map and then grab the the dictionary and then passing to this one so here we just did very passing the variable and so this variable can be that the parameter accepts at least a dictionary or a string to of a bypass to a vector data so right now we're using the second one so this one right now is a dictionary okay but you're welcome to provide at least of any Bounty boxes you want and then you need to specify the coordinates um because the value boxes if you see from this one for zero spells geospatial data usually you want to use some kind of a coordinates so this is just the latitude in launch Team by default the coordinates will be none right so in here a coordination none if this is none then you need to Bounty boxes you need basically need to provide the row and column number of the imagery so this is a little bit changing because if you give an image even virtual here it'd be difficult to know like uh what's the row number in the column number so the role in column number is counting from the top left corner right so it'd be like 100 rows uh uh 100 for example uh horizontal and also vertical is the column yeah and it's it's changing so I would recommend just using the coordinates and then just specify epsg 4326 so the wgs84 that's it so what we are telling the program is that okay I will want to give you two Bounty boxes and the coordinates is the wgs84 then I want you to segment the imagery the object from the Imaging based on those two bounding boxes and the output the D type is basically the output image data type so just run this one it should be pretty quick so it's done then you see we have the masked octave if you refresh on the left side you see this mask octave so it's done once you have the result then we can add the raster on the map so we're going to use the add roster and then this is the uh the bypass this is how I'm going to visualize the image using a color map and you can also specify the no data because this is basically a binary imagery one means the foreground zero means the background so and then you can also specify liname and then just run it take load so pretty easy right so now the rate uh the um yellow color represent the object that we extracted so you can zoom in here uh of course uh this is exactly what we want right so we do all the bounding boxes and then we want the algorithm to be able to extract the objects within the boundary boxes and if you turn this one you can turn the layer on and off you can just compare them right and you can also change the opacity uh it actually does pretty well right um if you don't want the bounding boxes anymore you can just click the button and then click them so now you can see this more clearly um and it's much better than the bounding boxes so now we have the profit object segmentation of this imagery but this is only have two objects there are so many trees and this is why uh if you will have the bounding boxes from other algorithm from other people in algorithms they already have the Bounty boxes then you probably want to use the automatic way just to provide the bounding boxes all of them and then you can segment all at once so what we're going to do here is going to use the geosystem from these five parts this is the one that actually created using the um text forms so the text form can generate the Bounty boxes so previously in the text from tutorial right and then you can generate for example the trees so I actually save these bounding boxes and then use as an input so if you come back to here this is the the file and we're going to download this Geo Json so this is a simple geosition with all the money boxes and let's take a look at what it looks like so this one this one basically creating a map we add the raster imagery that we downloaded and this is how you specify uh how you want to visualize the vector data and then use the add Vector file pass the style and whether you want to zoom the layer or not delay name so take a look so if you zoom out a little bit and this is what we have right for the input we have a lost imagery or we also have a vector uh data set with all the boundary boxes so if you want you can also explore uh here also the imagery right so with that now we can do the same thing because earlier we use the join tools to draw the Bounty boxes interactively but if you already have the Bounty boxes the polygon from existing data sources or from Deep learning output then we can use these bounding boxes to do the segmentation so it's the same step so the source code is exactly the same the boxes now again passing a geosystem so this geosition is basically this bypass it's the one that we downloaded in here if we double click right this is the Geo station so you can it can be a certified if you have a Geo package it can be anything that geopen does can open as long as jio payments can open you can just pass in the file pass so it can be even a HTTP URL to a geosition or to a zip file then geopen that's going to open directly that's fine so here we're going to use this jail station and then those coordinates is again epst 46 4326 now we also have an output so previously it's mask adoptive now we're going to have another one Mask two docties and the same thing we're going to use unsign integer08 because it's only a binary imagery so run this one and some of the coordinates might be out of the imagery boundary because for example in the lower right corner here you will see something like that so this one might take a little bit longer because there are more objects I forgot how many so basically it's over uh maybe a 100 object so it's going to take a little bit longer and you see it's done so must two dog teeth on the left side you see it's here so then we can add the result back to the map so take a look oops and here then we can tend we can change the layer right you see the trees are in red color and the bounding boxes in yellow color so you can turn the boundary auxes bounding boxes off and then you can take a look of course there are some of the trees here they will miss because the bounding boxes are not perfect so if you have add some more mounting boxes in here the tree will be extracted so the Bounty box is for hey export it from the text form so it's not like 100 uh accurate but you get the idea that you can use the bounding boxes and then use the segmentation just one line of code then you should be able to extract the object the mask from any imagery and so here the color map you're welcome to change to any May probably color Maps so for example we can do greens and then run this one again see the same layer so if you turn this one off right now it's then uh the green color so again it's very interactive so you're welcome to explore uh to use different color map uh different bounding boxes different imagery um so it's up to imagination and this is really usable because right now with this you can utilize existing data rather than starting from squids uh so if I can use points uh you can use bounding boxes you can use text prompts so now you have three ways they can use to do segmentation using the segment anything model um okay so that's what I want to show you uh in this tutorial I hope you enjoyed it and I will see you in the next one take care bye bye

Channel: Open Geospatial Solutions

Views: 4,314

Keywords: earth engine, geemap, geospatial, gis, google earth engine, ipyleaflet, ipywidgets, jupyter notebook, landsat, mapping, python, remote sensing, tutorials, jupyter, dataviz, gee, satellite, google earth engine tutorials, google earth engine python tutorials, geemap tutorials, python tutorials, ipyleaflet tutorials, folium, folium tutorials, geospatial data science, data science, programming, dem, leafmap, geopandas

Id: ayFgAj23BfI

Length: 21min 32sec (1292 seconds)

Published: Mon Aug 07 2023