Segmenting Satellite Imagery with the Segment Anything Model (SAM)

hello welcome to segment geospatial in this video I'm going to show you how to segment satellite imagery using the segment NEC model with just a couple lines called first go to the website same geo.gs hub.org you the link is also in the description and once you're here if you want you can leave the description so basically this package so you allows you to segment Center imagery pretty much anywhere using any satellite base maps and so originally the the package was uh inspired by the segment anything eel Ripple by Alexandra hanker ranker and so I built on top of his source code and developing a python package I also improve a lot and also provide notebook for you to try it out so you are welcome to go through documentation so the package allows you to download a map type from Time map services and create a geotif then we can use the geotif to do segmentation using the segment anything model then you can save the result also The annotation is geotif or also Vector data format I'll also provide a couple ways for you to visualize the result on interactive map or you can use a slider so here are some examples the pack is still under active development and here is a demo so for example after you do the segmentation you can create this kind of high level basically random color segmentation image and then can overlay on top of the original image to see what it looks like um so first let's go to if you want you can try out you can install this one on your computer but it will require GPU so if you only have CPU it's going to be pretty slow so I recommend that you come here and then examples right now we have three examples and it's still being developed I would like in this tutorial I would like to try to show you the automatic Mass generator so this is a new notebook that I just added yesterday um if you want you can scroll down you can take a look at the notebooks here it's pretty simple and straightforward so next I'm going to show you how we can get started click the open in Google color icon to open this one in Google collab you need to make sure that you log into your Gmail account once you're here the first step make sure that you change the runtime so you're going to click the right time and then change runtime type change this one to GPU don't use the CPU yeah it's very slow so once you click save you can check it out and you can upper right corner here you can click to connect or you can write any source code X to automatically connect this one to a GPU instance and okay so once it's done now we can first we need to install the packages so we need the segment geospace so the leaf map and the local type server for visualizing images on interactive map also allow you to for example visualize the base map so uncommon control slash your keyboard and then just one this one run anyway you should want this one and install all the necessary dependencies is to take a minute or two because this one if especially if I only local computer trying to install it's going to install pythons and if you have cooldown computer it's going to install the GPU version so it might take some time I'll probably call upper right corner here you can also see the rim uh basically because this is a GPU instance we have the ram almost uh 13 gigabyte we also the GP Ram I think it's about like 16 gigabyte also you have the disk and so later on we are when we are running the notebook using the GPU uh you will see the GPU uptick in here okay it's done then we can make sure that everything installs so safely then we can come on out the uh check this and check this one so next let's import the libraries so it's after you install the segment geospace so it's going to create under your site packages and it's going to call same zero and then this is the same jio cloud they were going to use second imagery and there are also a couple of other functions allow you to download images download files and also and visualized images so just import the libraries it should be a few seconds and once you import the libraries then we can create a map so we're going to use the leave map if you won't need more information about this map you can go to the website leafmap.org and then you should be able to find more information here I have tons of tutorials showing you how to do a data visualization using geospatial data so let's go back to here first let's create the interactive map so as you can see here we're going to create a map and then center around the latitude and longitude also the zoom label you can set the height basically the map height here and then we add the satellite base map but this one you can essentially use any basement you like right now we're just using the acetylvestment so this is the Google satellite businessmen without the label and the area here is around UC Berkeley so if you're familiar with the campus this is what it looks like and what we're going to do is to download the image in this area and then we can do the segmentation again if you want you can click this icon here and then you can change the basement if you want so if you click the investment icon this one shows you a tons of Base map so we are using this satellite base map here oops uh I think there's a bug here I might need to or duplicate um because I already added so let me today again um I think I need to fix the arrow so if you uncommon out for example and come out this one and then uh this is right now by default is the open Stream app so now if I change this one here oops uh of course you can change the turn the layer on and off and so let me uncheck this one click here and so here I can change the setup basement you will see here and um there are tons of other base Maps so you can use for example A3 base map imagery for example as long as you have any imagery you can use it so in this case we want to use the satellite base map and scroll down here um so it depends on what area you want to use you can use the uh the rectangle that I'm specifying so if you don't draw anything I'm going to use this area around UC Berkeley but you're welcome to select any air we you like so here oops it's a big okay here and how about this so I'm going to maybe center around this area and then let me zoom out a little bit okay so here I can draw a rectangle anywhere you like so for example I'm going to draw a area here okay and again you can zoom to anywhere you like once you zoom uh select the area and then it's going to you get the bounding box so the Bounty box is using this one here MW user Ry bounce if you want you can actually look at the coordinates so if I run this one you will see this is the low left corner right uh waist South north east north right so this is the bounding box with this a mounting box then we can download the images using the TMS to geotif so this one basically download all the map types and then Mosaic name is a single raster imagery so just run this one and the source this one is basically the base map so it's the set type base map but you're welcome to use any other base map that I show you in the drop down list you can set the zoom level the last of the number uh the larger the five size so you don't want to like make it too big because um you might don't have enough GPU to run the segmentation uh if you have any local images so you can also use specify the five parts to your local file uh image geotif if you want so once we have this then we can load the imagery to the map so here we're going to basically map layers negative one means the the layer the satellite base map layer um and we're going to turn it to force basically with a higher layer and then we use the m.a.laster so this is the imagery like you just downloaded and also the layer name so you can take a look at upper right corner here this is going to show you the imagery right so this is the base map if you want let me clear the ri right now we can see this is the image we just downloaded and this is a setup base map right so behind the scene and this one is the geotif it's Geo reference because you can see it's actually overlay on top of the map so the location is correct and so this is like gold beyond the original second test model that's only for uh regular images now we are doing that for seta images there might be some yeah sunlight doesn't show up you can run it again if you want to so once we have that then we can download the um we can start doing the segmentation but first we need to download the so-called checkpoint so the checkpoint you basically the model and we have all the uh model excuse me the model parameters and you can download this one to a computer so here I want to download the downloads directory but uh since we are running in Google collab I can just simply if we can download this one to here so what we can do maybe just simply use um you can uncommon this one even running a local computer uh you you are welcome to customize the bypass so I can just remove all this so this means that it's going to just download this one to my current working directory so the checkpoint we're gonna there are multiple checkpoints if you want you can take a look at the help documentation for this one we're just going to use the first model type and then the checkpoint we are going to use all the um default same parameters and so you can see the model is 2.56 gigabyte it's pretty big so even using it on your local computer it might take some time to download on Google call it it's pretty fast so you should download this one to the current directory here once we have the model then we can just hold the model so you see here on the left side with the model and then we initialize and once we initialize then we can start running the segmentation so the function you need to use is second uh same dot generate okay because this is right now we have this class this is the initiate the instance and then we'd want to join the function so the general function here if you have your mouse on the function name you will see the documentation right the source basically the image you want to do the segmentation and then you can say the output the foreground means by default the segmentation model we're going to segment everything so you're going to have all the objects covered in entire image if you just want some foreground it's not always going to be accurate but it's pretty uh Accu decent so other stuff you want to remove the background then you set this one by default to two batch means if the image is sometimes too big it doesn't fit into the memory you can set the base to two then it's going to subdivide the last image into smaller pieces and then do the segmentation on each individual one we also have the iglossian model it goes in corner kernel so basically after you do the segmentation sometimes if it's too noisy you want to extract the borders then you might want to set I erosion Chrono something like this three by three five by five so basically do that it grows and then so the object become smaller so then you see you're going to have the boundary Mass multiplier means that by default is going to segment the image into binary so it's going to be 0 and 1 and sometimes if you want to for example visualize that usually it's going to be RGB so it's going to do for example if you multiply by 255 then one become 255 so basically you have a raster of 0 and 255. so in that way it's going to show up uh in most uh image viewer otherwise it might be very dark the unique also this one is by defaulty too because as I said it's going to save the image into a binary so it's going to zero and one so everything will be one and then all the object will be just the same value sometimes it'll be useful to have uh unique identifier for each object and so the unique if you set two is going to starting from one two three all the way to the normal object so in this way it's better to easier to identify the object on imagery so these are all the parameters and again we just click one it only takes I think a couple seconds less than 10 seconds to run this imagery please be patient because on the right here you can also see the GPU so you will see when it's running right now we have GPU of 15 gigabyte so this is how much is being used and if you're using that excuse me running this one on a computer if you don't have enough GPU sometimes it might exit memory you see 13 seconds not too bad and once you have the do the segmentation on the left side here you see the mask dog teeth master teeth basically the segmentation result um again it just depends on the parameter you specify it might be a binary immediately 0 to 1 or 0 to 295 or it might be a unit identifier starting from one so once we have this one then we can use the soul mask and you can specify a color map so if you click it's going to want to show you right pretty nice it's pretty clean and this is what it looks like I have to do the segmentation right the background it's black color and so the unique identifier so you see some of those white color means basically when it's assigned the unique identifier and unique ID it's starting from including the area so it's going to calculate uh get the area of each object so the larger the area the brighter the color the smaller the area the the the the darker the color so this is actually pretty nice and this is just the gray scale if you want you can also uh so in a random color so this one here we use this function so a in an ends and is so The annotation so The annotation basically is just this object but assign a random color to it and we also set the axis off basically we don't want to see the um uh um X and Y axis also the alpha alpha means the opacity basically so if this zero Tracy is not it's fully opaque it's not transparent so the alpha is basically the transparency level also you can save the output so let's just run this one to see what it looks like we are going to basically export the image again earlier we already already have the mask but now we are exporting The annotation and The annotation is going to show above on top of the original set my imagery so you see here right we have all the color now it assigns a random color if you want some kind of transparency it's not like fully opaque you can change this one so I can do maybe three uh 0.4 and it's going to again install the original set time music and then show The annotation but it's going to somehow a little bit transparent so now you see it's you can see through the tweets right you can see the color but the object is not very clear it's up to you what you want to use again I can change it back to this one if you want also once we have The annotation so we have to for example right now we have the original set time music we have the mask basically the objects with um binary or unique values and then The annotation with random color so we can use the leaf map image comparison function this is the one that I just added yesterday so basically it's just for this package uh segment zerospace so allow you to compare the images side by side so we can just run this one it's going to take the set time usually you can actually pass in any image as long as the the same size so we're going to have the set image on the left The annotation on the right you can also have a label if you want so take a look at this right pretty nice and all right because earlier we use the uh set the alpha 2.4 if you set one then it's going to blow up fully um in uh fully OPEC but at least you can see compare right it's pretty cool also you have the label here seta imagery like the label the on the left also the label on the right and this allows you to compare imagery the result segmentation result easily this is just showing the result but uh because the result the segmentation and also the um the roster the second image everything is YouTube so they're actually um deal reference so you can overlay the images on top of the interactive map I think it's better to change this one back to one so that is it looks better when I overlay on top because you can change the opacity directly on the map um so it's going to re-output The annotation it's a fully opaque so it looks like this then we can run this one here so again we're going to add this annotation on top of the standard imagery so now we take a look all right so this is what it looks like right now on the map you can turn the oops uh I think I need to um sometimes study the map doesn't work very well let me let me do it one more time it's a little bit buggy on Google collab but if you do that on your local computer it should work better so let me recreate the map and then I can add the imagery so oops let me run this one again when we create a map and then we add the base map like this here so in this way the toolbar icon shows up so and then later we can use this one to turn the layer on and off oops yeah so we have the map now and then come back to here there's one this one okay so now you have this one and you can use the layer control here like the mask the image and the original uh select base map right so here if you want you can turn the layer on and off and it should be interactive so if you want you can zoom into anywhere you like for example here low left corner right so we have the segmentation result we also have the set animatory the layers Santa doesn't show up so again it's a little bit buggy on here it's basically the visualization but you can so I download imagery and then try this one on your local computer it should be much much better so again right so this is basically just a segmentation right now it doesn't have any label yet so I'm still working on editing a feature so if you want you can uh start the repo and so now I'm working on this one trying to allow user to for example can select a point so I can um on the internet map you can select the point you uh the object you want to extract and then the model automatic extract for example I place a point here it's going to export uh objects and Welding this one and it is going to guess like different objects so this is going to Output three mask and then you can pick which one you think is the correct object again it's still work in progress and you're welcome to try it out and after you do the segmentation you can also set uh convert the raster to track the data so here right this is just a segmentation with a random color and you can convert The Mask basically the unique identifier uh the object which you need identified to a vector data and you can refresh here right it can go into convert to and jio package or you can do for example a stepfather so SSP you can also export to Geo station exchange so you can export any Vector data supported by jio pandas for example geojson and under the hood is being exporting to a geodata frame and then convert the geodata frame to any Vector data format you like so this is the examples using the all the default parameters so you can simply download imagery for any location you like try this or small area don't try too big and then you can run the the code to export to do the segmentation and the same model there's some parameters that allow you to fine tune somehow but it's not like fully customizable but at least you have some problems if you want to customize how many points you want to join with 32.32 points by default and um drop the layer now scaling and also I think the most tips will be this one minimum Mass region area so because otherwise you when you do the segmentation you're going to join a lot of random sample noise so if you want just on last object I want to reduce the pay so and paper effect you can increase this number so by default is 100 pixels and with this you will see it's pretty much the same as the previous example right except right now we add this one so the same argument so now we can specify these argument to fine tune the model and then everything else pretty much remain the same so we're going to we need to re-initialize the model using the same model type and also the checkpoint but this time we're passing this same arguments and once you have this it's the same procedure so I'm going to generate the imagery again it's going to take about 10 seconds and then you can visualize you can show The annotation you can do the same image compression and lastly let's another image function here overlay images this one doesn't doesn't work on Google collab but if you trade it on a local computer you have something like this so you can join your image using a map probably and it's going to have a slider so that you can also change the opacity if you already use the map you probably don't need this because you can add the two data layers on the map and they can change the opacity interactively but if you don't want to use deepmap then this function might be useful for you you see it's still running because we fine-tune the model we provide the parameters it's take a little bit longer so this time it takes uh it took 40 seconds and once you finish uh do the segmentation you can join the result so this one now looks like this it's a bit different from the one you saw Here If You Can notice that right I think the object right now is more fine scale so you have more objects compared to this one and this one here right so you can download images you can do the compassion uh if you want again now we can show The annotation uh similarly you can use the image compression to solve The annotation if you like so you can change the slider so this time right now is The annotation uh two so I'm going to show for example annotation to here so the image compared images side by side and we'll we also have two masks in here right Mass one and also Mass two so you can actually also use the image comparison to anyway so again so this is the new segmentation using the Y2 model and the downside you cannot zoom in you know zoom out you can only use this one to do a slider to do a quick check but if you want more details you want to tune into the map then you can use this way as probably is better so you can zoom in and zoom out you can turn the layer on and off so but if it is on Google collapse sometimes the Thailand I take some time to show up okay so that's pretty much about this tutorial uh shows you how to download imagery space map as a geotif and then do the segmentation and this is only the first step right so I have to do the circumstation you need to assign the label the class right now it has no idea about what its object it is and this is what uh the piece today that I showed you earlier that will allow you to pick the feature and then do the segmentation so in that way you know it is a building is it a tree or something else so hopefully you'll be finished implementation in the next couple days and then you can I will produce another create another video tutorial showing you how to do that but for now at least you can get a nice segmentation um and this is usually much better than traditional pixel based classification and it takes a little much longer time this is pretty much using no training data you just input imagery and then initialize the same class then you can just do the segmentation and then you can fine tune you can change the parameters if you like but at least it looks very clean pretty decent okay so that's all for this video I hope you uh enjoyed it I will see you next time take care bye

Channel: Open Geospatial Solutions

Views: 28,134

Keywords: earth engine, geemap, geospatial, gis, google earth engine, ipyleaflet, ipywidgets, jupyter notebook, landsat, mapping, python, remote sensing, tutorials, jupyter, dataviz, gee, satellite, google earth engine tutorials, google earth engine python tutorials, geemap tutorials, python tutorials, ipyleaflet tutorials, folium, folium tutorials, geospatial data science, data science, programming, dem, leafmap, geopandas

Id: YHA_-QMB8_U

Length: 25min 46sec (1546 seconds)

Published: Mon May 08 2023