so this is the project in which we will be working
today, on today's tutorial, this is a background removal web application and let me show you how
it works, I'm going to upload an image from my computer I'm going to click over here and then I'm
going to click on remove background and you can see that we have completely removed the background
on this image, so this exactly the project in which we will be working today, on today's tutorial, we're
going to work with python streamlit and the segment anything model, so let's get started. Now let me
show you what are all the steps we will need to take in order to complete this process let me
give you a very high level description of today's tutorial so the first step we will need to take
in this project will be to create a function which will be called remove background and this is the
function we are going to use every time we want to remove the backround on our images and in order
to make this function we are going to use Python the segment anything model and also Google collab
I'm going to show you how to create this function from a Google collab then the next step in this
process will be to deploy this function to serve this function through an API and in order to do
so we are going to use modelbit which is an amazing platform to build and deploy machine learning
models and I'm going to show you exactly how to do it later on on this tutorial, then the next
step in this process will be to create the front of our web application and in order to do so
we are going to use Python and streamlit and the last step in this process will be to call
the API from the web application right we are going to call this API we have created over here
from our web application and that's going to be pretty much all in order to create this background
removal web application and now let's get started with this tutorial and the first step will be to
create this function remove background so let's go to my Google collab and let me show you how
to create this function so the function we are going to create it is called remove background
and this function is going to take an image as input and it's also going to take the X and
Y coordinates of the subject we have selected when we clicked on this image and this is the
function we are going to create in this step of this process, and in order to make this function
I am going to write all the steps we are going to take in order to build this function so the first
step will be to install all the requirements all the python packages we need in order to make
it work obviously the segment anything model will be one of our python packages but we will need
to install other python packages as well now the next step in this process will be to create
the segment anything model predictor we will be using in order to make inferences in order to
make predictions this is a very important step then although we want to create a very generic
function that just takes any image and any X Y coordinates we input into this function we want
to create a very generic function we obviously want to test this function with a given image
and with some given X Y coordinates right so we are going to load image and we're also going
to select the X Y coordinates we are going to use... to test this function right we are just
going to load an image and we're going to select some X Y coordinates, then the next step in this
process will be to use the segment anything model predictor we created over here on this image and
on this X Y coordinates... to get location of subject right and I'm going to show you exactly how to do
that later on and then once we have... once we know exactly where our subject is located the only
thing we need to do is to remove the background right so these are exactly the steps we will need
to take in order to make this function and then once we have made everything else we have made
everything that's in this process the only thing we will need to do is to wrap it up as a function
right because we're just going to take all these steps we're going to execute all these steps one
at a time and then at the end we're just going to take all the code we created and we're we're going
to wrap it up as a function, I have a type over here this is background... something like this and
then something I'm going to do now so everything is much more organized everything is going to be
much better organized I'm going to create a new cell for each one of these steps so I'm going to
do something like this this one over here this one over here this one over here over here and over
here okay... so now everything is going to be much better and let's just get started with the first
step in this process which will be installing all the python packages we need in order to make this
function so the first python package will be the segment anything model so the only thing I'm
going to do is to copy and paste this code we have over here right pip install and then... this
GitHub repository over here and obviously we need to start this sentence with an exclamation
point because that's the way we install python packages from a Google collab and then we also
need to install torch torchvision opencv and numpy so the only thing I'm going to do now is to
execute this cell okay, so we have installed all the python packages we need in order to make this
function and now the next step will be creating the segment anything model predictor we are going
to use in order to make inferences and this is how we are going to do, I'm going to import from
segment anything... I'm going to import sam predictor... and also sam model registry okay so we
are going to make these two Imports and then let me show you this GitHub repository which is the
segment anything model GitHub repository and let me scroll down until this section over here which
is exactly the code we need in order to make this predictor so the only thing I'm going to do is
to copy and paste this code over here and you can see that we have two variables one of them
is the model type and the other one is the path to checkpoint so let's... let's get back here to this github
repository and now I'm going to scroll down until this section over here which is model checkpoints
you can see that we have three different models and basically the difference between these models
is their performance and their execution time this model over here has the best performance so
it produces the most accurate results and then this model over here is the best execution
time, it produces the fastest results right we are going to have an inference in a shorter
amount of time this is the best execution time and this is exactly the one I am going to use in
this tutorial so the only thing I'm going to do is to copy this value over here and I'm going
to paste it over here... and then when it says checkpoint and the path to checkpoint what I need
to do now is to get this value this URL... and I am going to do something like this I'm going to call
wget... and I'm just going to download this file we have over here and I'm going to download this file
into this notebook but before something I can do in order to make it look nicer, I'm going to
define a variable which is model path and I'm going to do something like this model path will will be
something like this and then if not os path exists model path... then we are going to download this file
right it's going to be much better if we do it like this... because just in case we need to execute
this cell again then we are not going to download the file again right so the only thing I'm going
to do now is to... edit this value and then I'm just going to press enter... you can see that we are
downloading this file and that's pretty much all, so now let's continue so remember there are
three different models we can choose from and their difference is basically their performance
and their execution time and in this tutorial I'm going to use the one with the best execution time
now moving to the next step over here we need to load the image and the X Y coordinates we are going
to use in order to test the function, now let me show you the image we are going to use in
order to test this function this is the image we are going to use and the only thing I'm going
to do... I'm going to do something very similar as we did over here with this file we downloaded I'm
going to copy and paste this code over here but I'm going to replace the URL by this URL we have
over here so this is going to be something this and now the image name will be test dot jpg and
this will be image path something like this okay copy paste... so this is going to download an image
and that's the image we are going to use in order to test this function but we also need to select
the X Y coordinates we are going to select we are going to click from this image and in order to
do so let me show you what we are going to do I'm going to download this image into my local
environment and I'm going to show you how to select the X Y coordinates so I'm going to open
a terminal and I'm going to say something like wget and the image URL, then the image has been
downloaded and let's me show you the image over here... this is the image we have just downloaded
and I'm going to open this image with image magick which is a very interesting software in
order to work with different image processing related projects and now I'm going to click on a
section of the image for example over here where our subject is located over here and this is the
coordinates 528 606 so the only thing I'm going to do... I'm going to define two variables one of
them is X and the other one is Y, X will be 528... 528... and then Y is 606, image magick is an amazing
software, an amazing software to work in computer vision so go ahead and download image magick as
well in order to work with your images and in order to get the XY coordinates of a given point
right it's a very very powerful software now let's continue so we have completed this step over here
which is loading an image and also getting some X Y coordinates we can input into the function
but obviously if we want to load the image if we want to load the image into memory we are going
to do something else we are going to import cv2 and we're going to say something like image is
equal to cv2 imread and then image path... and that's pretty much all now I'm going to execute this
cell... and now let's continue and now it's the time where we are going to use this segment anything
model predictor in order to make inferences in order to make predictions and this is how we are
going to do I'm going to call mask predictor set image... and then I'm going to input the
image we have just loaded over here... okay then I am going to call a function which is mask predictor... predict... and I'm going
to input three arguments... point coords... this is going to be a numpy array... and I'm just going to input the X and Y
coordinates we defined over here and then I am going to define two additional variables...
arguments... one of them is point labels this is also going to be a numpy array right and I need to input
the value 1 and then the last argument will be multimask output and I'm going to set it
in True then this is going to return three variables which are masks, scores and logits...
we don't really care about the scores and the logits we we're only going to use masks but we can
just go ahead and define these two other variables as well let me do this a little prettier something
like this and it seems we have an issue with this variable name and I see we called this variable
predictor only predictor so I'm going to over here and I'm just going to call this variable
predictor okay so now I also need to import numpy as np otherwise this is not going to work so
import numpy as np okay now I am going to take the output we got from this function, masks, and I am
just going to process this variable in order to get exactly what we are looking for which is the
location of the subject right I'm just going to write it down first and then I am going to explain
it so I'm going to write it down first this is going to be something like this... and now let me
explain all these instructions we have over here and before I do so I invite you to take a look
at one of my previous videos where I show you a super comprehensive course of opencv with python and
one of the lessons in this course was about image processing was about how to work with images in
opencv and other lesson in this course was about colorspaces so I definitely invite you take a
look at this course and I invite you to take a look at these two very specific lessons because
from now on we are going to work a lot with image processing and with colorspaces so it's very
important you are familiar with how to work with images and how to work with colorspaces for
everything that comes from now on, let me give you a very quick and a very high level description of
what I am doing over here remember we are getting all the predictions all the inferences for the
image we input into the function which is this one right and remember the output is not going to
be one individual image, one individual mask but we are going to have many many masks remember we set
this variable which is multimask output as true so we are going to return many many masks, let
me show you an example these are all the masks we are going to get once we call this function
you can see that these are many many many masks and also we we are going to have many other masks
which are a combination of them for example if we click over here as we have just clicked a couple
of minutes ago then we are going to have one of the masks which will be the entire person but we
are also going to have another mask which will be this section over here right everything that's on
this blue color over here and if we select this section over here we are going to have the person
the entire person but we're also going to have the hair right and if we click on the person's face
then we are going to have the entire person but we are also going to have the person's face so
from now on we need to make sure we are working with the mask we are interested in which is the
mask of the entire person right and this is how we are going to do I'm going to define a new
variable which is result mask and then we are going to iterate in all the masks we have received
for this very specific location right we don't know how many many they are but we are going
to iterate in absolutely all of them and then the only thing we're going to do is to merge all
these masks together and by doing so we are going to produce the mask of the entire person right we
know one of these masks is the mask of the entire person and then we will have other masks which will
be like... smaller, but if we do something like this if we just merge everything together then we will
have exactly what we are looking for which is the mask of the the entire person just go through
this code once and again until you probably understand exactly what we are doing over here,
we are producing many many many many many masks but we are just merging everything together so
we produce the mask of the entire person which is exactly what we are looking for so this is
exactly what we are doing over here and now I'm going to press enter okay we have very small
mistake this is actually something like this... we need to say something like this and now I think
it's going to be just fine, okay now let's continue now we have the next step in this process which
will be removing the background from our image so I'm going to write it down first I'm going
to write all the steps all the instructions we need to take in order to remove the background and
then I am going to explain it, now let me explain these instructions over here remember it's very
important you are familiar with colorspaces and you're familiar with image processing because it's
very important for everything that comes from now on, now let me show you this over here remember
this is the image we are working with this is the image we are going to use as an example and this
is the mask we would like to produce from this step over here right, we should have a binary image
where absolutely every single pixel is either white or black and all the white pixels are the
location of our subject right, let's... let me show you something, if I make an overlay something like
this you can see that the white pixels in this other image are exactly the location of the subject we
are interested in this is how ideally the result mask variable looks like right how this... how
this mask how this image looks like for this particular image right this is only to give you
an example... and now remember you have to be super super clear on how colorspaces work and how...
images work, because until now we were working with an image of three channels right this image
we are using over here has three channels which are blue green and red right we're working with
a BGR image and this mask we have produced over here is an image with only one channel right
this is a binary image with only one channel and absolutely every single pixel is either white
or black so in order to remove the background of our images we are going to produce a new image
with four channels and the first three channels are the image we were working so far right the
first three channels are going to be exactly the same information as this image over here and
then the fourth channel will have the information of the mask we got from the segment anything model
right so this is why we are creating this variable which is called alpha channel and alpha channel
will be this additional channel we are going to create and this is the channel which is
going to have all the information related to all the pixels of all the background in our images
right, and all the black pixels in this image are the background right so if we want to remove the
background the only thing we need to do is once we create this new variable which is alpha channel
we are going to set this variable to zero in all of those pixels for which, this variable over here
this mask we have created, for all of the values in which this mask is zero as well right we are
going to take all the values of the background and we're just going to set this alpha channel as
zero in all of those pixels that's the only thing we are doing over here and then the only thing
we need to do is to create a new image with four channels, the first three channels are going to
be exactly the same as the image we are going to use as input and then the fourth channel will be
this alpha channel we created over here and this is going to take care of removing the background
right we are not really removing the background because the background is still there but we are
not really going to see it right we are going to set the background as a transparency right
that's exactly what we are doing over here, so we're going to use the information in this new
variable which is alpha channel we're going to use this binary image this binary mask in order
to control the transparency in the new image we have just created and this going to be pretty
much all in order to remove the background in our images, so we made it, that's pretty much all
in order to create this function actually these are all the different steps we need to do in
order to create this function and now the only thing we need to do is to wrap everything up as
a function, the only thing we need to do now is to take all this code we have created over here and
to create a function from this code so this is how I'm going to do I'm going to copy and paste this
function over here and I'm going to delete this cell over here because we don't need it anymore
and now the only thing I'm going to do... I'm going to remove this 'pass' and now the only thing
I'm going to do is from here... I'm going to copy paste... and then we are going to copy and paste... this over here... and then we are just going
to return result image and that's going to be pretty much all now let's see if it works so I'm
going to press enter and now the only thing I'm going to do is... I'm going to copy... this code
over here... and now I am going to call this function... going to input image and then X Y... this is going to be result image... and then let's see how it
looks like so I am going to import maplotlib... as plt... and then the only
thing I'm going to do is to plot... result image and let's see
if everything's okay so plot imshow... and then plot. show okay now let's see what
happens... and this is pretty much all this is the result so you can see that everything is okay
there is a very small issue related to the colorspace in which we are visualizing this image but
other than that everything is just fine, in order to make it right the only thing I'm going to
do is to change the colorspace like this cvt color result image and then we are going to call
cv2 color this is bgra to rgba let's see what happens... and you can see that now everything
looks just perfect so we can continue, now you can see that we have completed this process
we have made this function we have created this function which is remove background everything
is working just fine so we are ready, there's a very small thing we should do now because you
can see that this function over here receives an image and this image is a numpy array right but remember
we want to create an API from this function so we need to do something because we cannot post a numpy
array through this API, we will need to do something and let me show you what we need to do we are
going to convert this image which currently is a numpy array, we are going to convert it into
bytes and then we are going to encode this new object with base64, we are going to create a
base64 encoding and this is what we are going to input into this function so let me show you, I'm
going to change the variable name this is going to be something like image base 64 encoding...
encoding something like this and then we are going to convert this object into bytes first
and this is how I'm going to do I'm going to import... base64... and this is going to be base64
dot b64decode... we are going to input the image and this is going to be a new variable
which will be image bytes and now we need to make an additional conversion because now we
want to take this object which is a sequence of bytes and we want to make this object
into a numpy array let me show you how we're going to do, I'm going to call another function which is
cv2 dot imencode... and we are going to call np from buffer... we're going to input this object over
here... and then dtype np unsigned int 8 okay and the only thing we need to do now is to
do cv2 imread color okay and this is going to be the image right so you can see that the
only thing we're doing is we are taking this object which is going to be encoded in base64
we are decoding this object we are creating this new... this new variable which is image bytes
and then we are just converting this image bytes into a numpy array and from now on everything
is going to be exactly the same now we need to do an additional change because the output
is not going to be a numpy array but it's also going to be an object which is going to be encoded
in base64 so let me show you now we're going to do exactly the same but we are going to do
the opposite operation right let me show you now we're going to call cv2 imencode we going
to encode it as a PNG and this is going to be the result image... then this is going to be result
image bytes... we're going to convert these objects into bytes first and then we are going to call...
something like this just to make sure this are bytes... and then we need to do something like this
base64 dot b64encode... we're going to take this object... and we are just going to encode it and
we also need to say something like this UTF 8... okay and this is going to be... encoded base64... okay okay this should work
just fine so we are taking the image we are converting this image into bytes and then we
are encoding this bytes this new object into base64 and this is what we are going to
return from this function... now just to make sure everything works well just to make sure
everything is okay we are going to test this function again and now obviously we will need to
do something here because now we are not going to input this object over here which is image
but we are going to encode image into base64 so I'm just going to copy and paste and I'm
going to change it... over here this is going to be image bytes... image bytes... image bytes... image
bytes... and this is going to be image bytes encoded base64... and this is going to be the
input of our function and now we need to do something as we did over here because we are
going to receive something which is encoded in base64 and we are going to decode it, so this
is going to be over here this is going to be result... image bytes... and then result image bytes
over here... and then this is going to be result image... okay now let's see if it works okay there's
a very small mistake I think there is an issue over here... actually this is going to return two
variables... this one we are not going to use it and this one is image bytes so we need to change
it over here as well... something like this let's see now... oh I see this is to bytes... I'm going
to change it over here as well and now everything should be okay let's see now...
we have another mistake... this is... there is a very small typo over here... this is uint8 and
obviously we need to fix it over here as well... over here... okay I'm going to press enter again
in the function I'm going to press enter again over here, and now we have another mistake
because over here this is decode this is not encode right because we're going to
receive an image which is encoded in base64 and we are going to decode it, let's see now, and
obviously we have another issue... this is imdecode... imdecode... and we have the same mistake over here
this is not imencode but this is imdecode okay and now we are almost there we are getting the
image but there is a very small mistake over here because we are not getting the image in four
channels but we are getting only three channels so if if we're getting only three channels we
are just going to get the original image right we are going to change this value over here now
it's going to be imread unchanged and this is going to be just fine now we're going to get a four
channels image and this is going to we just fine exactly what we need and you can see that now
everything looks perfect everything looks the way it should so it took a few minutes but finally
we made it and now this function is ready in order to move to the next step in this process which
is deploying this function using modelbit, modelbit is an amazing platform you can use to train
and deploy machine learning models on gpus and modelbit is sponsoring this video just to make
it clear and now I'm going to show you how to deploy this this function using modelbit, this
is going to be a very straightforward process and it's going to take only a couple of minutes, let
me show you, I'm going to create a new cell over here and the only thing I'm going to do is to
install a new python library which is modelbit and this is the python library we are going to use
in order to deploy this function okay so we have installed this new python package and you can see
that now we need to restart this runtime so I'm going to click here and that should be enough
in order to restart this runtime so I'm going to create a new cell and this is what I'm going
to do I'm going to import modelbit and then I'm going to call modelbit . login because we need to
connect with our account in order to deploy this function right and in order to do it is
going to be super easy the only thing I'm going to do is to do something like this I'm going to
press enter and this is going to make sure to connect this instance environment this notebook
environment with my modelbit account and this is the only thing we need to do in order to deploy
this function I'm going to click here, this is going to take me to my account... and everything
is ready you can see that your kernel has been connected to modelbit and now let's continue and
now the only thing we need to do is to call mb dot deploy and the function name which is remove
background so I'm going to say something like this remove background, there's a very small thing
we should do first because if I scroll up... remember now we have restarted this runtime so
now all the variables have been lost so we need to execute this cell again before we continue and
I'm going to do something first because we are going to execute this cell where we have defined
this function but we need to do something else I'm going to copy this code over here which is where
we have defined the predictor... I'm going to paste it over here and then I'm also going to import
cv2 because we are using cv2 over there... and then that's pretty much all I think the other library
is base64 which is over here and then everything else numpy as np I'm going to import numpy as well...
numpy as np... and that's pretty much all so I'm going to execute this cell over here and then I'm going to
scroll all the way down and I'm going to execute this other cell about deploying this function
and what's really amazing about modelbit is that it's going to take our function, all the the
function definition and all the code which was in that cell and it's also going to take absolutely
all the objects and all the files and everything it needs in order to execute that function and
it's just going to upload everything and it's just going to work just fine right because if
you look this function closely you can see that we are using this predictor which is not anywhere
in this function we have this predictor over here and this predictor is using this file right it's
loading this file from our hard drive... from from our disk... so there are many objects and many
files which are needed in order to execute this function and modelbit is just going to take everything
and it's going to upload everything so everything is going to work just fine now if we go back
to modelbit and I refresh... you can see that now I have this other function over here which is
remove background and now everything is ready in order to use it if I go to API endpoints you can
see that this is the API endpoint we created and if I go here to source code this is the function
we have deployed and this is the... this is the file we need in order to execute this function
so everything is just ready in order to execute this function in order to call this API and we
can just continue from here and now let's see if everything works well the same way we did over
here... when we tested this function now we are going to test this API and let's see if it works, so I'm
going to copy this code which is going to be very similar... we are going to
define a new variable which is.... API endpoint... this is going
to be this endpoint over here... then we are going to define an object maybe
I can do it over here... over here, this going to be API data, this is going to be a dictionary, we are
we going to have only one key which is data and is going to be a list, we need to input three
values one of them is this one and then the X and Y coordinates so I'm just going to type X
and Y okay and now the only thing we need to do is to call this API and in order to do so I am
going to do something like this, response will be equal to requests . post API endpoint and
then json API data okay obviously I need to import requests... something like this and then the response
will be over here we're not going to call this function anymore... maybe I can do something like
this... if I'm not mistaken this is going to be something like this response json data and if I'm
not mistaken this is going to work just fine so we are defining the API endpoint over here then
we're just calling this endpoint we're making this post request... API data everything looks just
fine and then the response will be the image which is going to be encoded in base64 we're going
to decode it and everything is going to work just fine let's see if it works so I'm going to
press enter and let's see what happens... and this is what we got so everything is working just fine
so we have deployed this function using modelbit and we have just tested this API and everything
is working just fine so we can continue with the next step in this process, you can see how simple
it is to deploy a function using modelbit, it took more time to test the API to make this code over
here, it took more time to test the API than the time it took to deploy the function right so
it's amazing, now let's continue with the next step in this process which will be creating the
web application and in order to do so let's go to pycharm... so this is a pycharm project
I created for this tutorial I have already created this project I have already created the
virtual environment and I have already installed all these python packages we have over here so
before we start on this tutoral before you start on this part of this process please remember to
create a virtual environment and to install all these requirements over here now let's continue
so the first thing we're going to do in order to move one step at a time is to write all the steps
we are going to follow in in order to make this web application and the first one will be...
setting up the layout we are going to use in order to make this web application right remember
we are going to have some widgets we are going to have some images and so on and everything is
going to be in a given position so we are going to set the layout in which everything is going to be
located then we're going to create a widget which is the file uploader this a very important widget
because otherwise we will not be able to upload a file, then we are going to read the image the user
is going to upload... read the image we're going to say something like read image, then we are going
to create a couple of buttons... then we are going to visualize the image so something like visualize
image... and then we are going to click on image get coordinates and call the API, so basically those
are all the steps we are going to follow in order to complete this process and in order to be more
clear on exactly what we are going to be making now let me show you this diagram over here, if I go
over here this is exactly what we will be making now, you can see that this is the user interface
we have this widget over here which is the file uploader then we have these two buttons which
are original and background removed and then we have this other widget over here which is where
we're going to visualize the image and basically if you remember from the intro this is going to be
something like this the user is going to upload an image and then he can... he or she can click on this
button which is background removed and this is going to remove the background of the image and once the
user clicks here we are going to call the API and we are going to get the image with its background
removed so this is exactly what we are going to be doing now and a very quick note we are going
to take care of making everything that's related to this part over here... and then at the end I'm
going to show you how to change the background and we are just going to put this background over
here but the background is only a detail so we are not going to mind this background we're going to
do it at the end for now we are going to focus on this part over here so let's go back to pycharm
so let's get started and the first step will be importing streamlit as st... streamlit as st... okay
and now we are going to start over here which is setting up the layout and we're going to call st
set page config, and we're going to set layout to wide... okay and now we are going to create two
columns and we are going to do it like this we're going to call st columns... two and this
is going to be col01 and col02 because remember we have the file upload widget and the
image and the buttons and everything else we had everything on... on the right side right
and everything on the left side was completely empty in the left side the only thing we had
was the background so we are just going to work with this col02 we are going to make
everything over here so this is going to take care of making the layout exactly as we want
it, now let's continue, and now we are going to create the file upload widget which is going
to be something like this col02 dot file uploader... this is going to be empty and
then the type will be something like jpeg... jpg... or png okay now let's continue now
I'm going to make another import which is from pillow import image and obviously
I need to make this equal to file this is going to be a variable which I'm
going to call file and if file is not None... then I'm going to say image
open file and I'm going to convert it to RGB okay and this is going to be
a variable which I'm going to call image... okay now everything should be over
here okay now let's create the buttons we are going to use in order to remove the
background of our image and in order to visualize the original image right we're going
to have two buttons one of them is going to be original and the other one is going to be
remove background so let's see how we can do that, now I'm going to say something like
col02 dot columns two right we are going to do something very similar as we did over here
but now we are going to call col02 instead of st so this exactly what we are going to do
now and this is going to be col1 and col2 okay and now let's continue so I'm going to
create the buttons like this if col1 button original use container with equal to true I'm going to make it 'pass' for now I'm
going to write the code later on and then let's do the same for the second button something like
this if col2 button this is going to be remove background and I'm going to set an additional
argument which is going to be type equal primary because we want to make this button of
another color and that's how we're going to do it, primary okay,and that's pretty much all
is going to be 'pass' as well and now the only thing we're going to do is to continue visualizing
the image and everything is just fine let's make sure everything is working properly and let's
execute the code as it is now, so I'm going to the terminal and I'm going to type something
like streamlit run main . py and it seems we have an error it seems I didn't install pillow
so let's see what's going on let's see over here maybe if I do another import which is from PIL
import image I think now everything is going to be just fine let's do again... okay now everything
seems to be just fine I'm going to upload an image for example this one and now we are visualizing
two buttons so everything is just fine for now let's continue, and now it's time to continue with
this step over here which is visualizing the image and in order to visualize it I am going to do these
two steps at the same time I'm going to visualize the image and I'm also going to do everything
that's required in order to click on the image and getting the coordinates and in order to do
so I am going to use this python library over here let me show you this is the python package
we are going to use this is the GitHub repository this is the code we are going to use in order to
visualize the image and in order to get the image coordinates once we click on the image right, it's
called streamlit image coordinates so let's get back over here and this is what we are going to
do obviously the first step will be to make the import so from streamlit image coordinates
import streamlit image coordinates as im coordinates... everything is okay and now the
only thing we need to do is something like this so I'm going to do it over here I'm going
to do it after we create these two columns but before we define these two buttons over here
this is the best way to do it and the way I'm going to do it is by using a placeholder so I'm
going to call col02 dot empty and this is going to be placeholder zero because we're going to create a
couple of additional placeholders so we're just going to call this this one placeholder zero and
then we're going to say something like this with placeholder zero... we are going to call im
coordinates... we're going to input the image and then we are going to call the output value
and then the only thing we're going to do is if value... if value is not None... then we're going to
print value to make sure everything is okay, okay so let's see if it works, so let's get back here
and I'm going to refresh, I'm going to upload an image, you can see we're getting the buttons
we are getting the image something is going on because we are not really seeing the entire
image but we are... we need to do something we are going to do something later on but let's just
test the functionality of clicking on the image and getting the coordinates so I'm just going to
click over there and if I get back to pycharm and I go to the terminal you can see that now we are
getting the coordinates so everything seems to be working just fine and this is very important
because this is like... 90% of our job is done right 90% of everything we need to do is ready
because we have the image and we have the image coordinates right the only thing we need
to do now is to call the API and that's pretty much all so we are almost there to complete this
process let's see if it works if I click on the image again... and you can see that we're getting the
same... we're getting other coordinates so everything is okay now let's continue with this process
and let's see how we can fix this issue we have because we are not visualizing the entire
image but we are just visualizing part of this image right we will need to resize this image so
we make sure it fits the container exactly right we want to make this image to fit exactly this
container over here so this is how we're going to do, we are going to use another python library
we are going to use this python library over here which is streamlit dimensions and by using this
library is going to be super super easy to know exactly how we need to resize this image over here
so let me show you I'm going to say something like this screen dimensions this is going to be... I
am going to import from streamlit dimensions import st dimensions and I'm going to call st
dimensions and I want to know what's the size of the entire screen right because then we are
just going to take half of this value and then we're just going to resize the image by half
of this value so let's take one step at a time and I'm going to print screen dimensions and let's see
what happens so let's get back here and I'm going to refresh... I'm going to upload an image and now if
I go back to pycharm and if I go to the terminal you can see it says 1760 so the only thing I'm
going to do is to divide this value by two and this is the value at which we need to resize this
image width in order to make sure it's going to fit entirely this container over here so this is
what I'm going to do, I'm going to something like this now we know what are the screen dimensions we
don't really need this library anymore and now I am going to resize this image doing something
like this image resize the new width will be 880... and then the new height will be something like
this image height multiplied by 880 and then divided by image width, okay and that's going to be pretty
much all in order to resize this image, let's see if it works so we take it one step by time if
I go back to my browser and I refresh, now I'm going to upload an image and you can see that
now we are visualizing the image perfectly right we are resizing this image and now the image
fits its container entirely right everything is just perfect so we can continue with the next
step in this process, okay let's continue and we are almost almost there we have almost completed
this process and now let's continue by adding the functionality of these two buttons over here and
this is why we are using placeholders now you're going to see why we are using placeholders because
this is what we are going to do I'm going to call placeholder zero . empty and this going to take care
of removing... the image we have visualized over here right, so as we are visualizing many
different images we're visualizing an image over here we're going to visualize another image
when the user clicks on original we're going to visualize another image when the user clicks on
remove background we're going to visualize many different images we need to make sure we are
removing the other images before we visualize the new image right let me show you so when the
user clicks on original this is going to remove whatever we have visualized over here we're going
to remove this placeholder and then we are going to say something like this, we're going to create
a new placeholder so I'm going to say something like this, this is going to be placeholder
one and then we are going to do something like... very similar as we did over here but we
placeholder one and the only thing we're going to do is call col02 dot image and we are going
to visualize the image and we're going to use it... let's see if I have it over here I think I
don't use column width equal to true okay so this is going to take care of visualizing the
original image which is exactly what we want when the user clicks on original and now let's do this
other part over here which is the most fun part the most interesting part because this is when the
user is going to remove the background once it clicks here so I'm going to start doing something
like we did over here, okay but now this is going to be placeholder two and now we're going to do with
placeholder two and we are not going to visualize this image but we're going to visualize another
image and all the code we need in order to call the API and in order to get the image we get as a
result of executing this API, we already have this code because it's exactly the same code we use
from the Jupiter notebook in my Google cab so let me show you, so let's get back to my browser
let's go back over here the only thing I'm going to do is to copy and paste all this code over
here, we are taking an image which is a numpy array and we are returning something which is also a numpy array
so I'm going to copy and I'm going to paste it over here... I'm going to fix the indentation first...
I think everything is... no everything is okay okay so I'm going to make these imports opencv...
we don't need this import anymore so import cv2... import numpy as np and then we also need
to install... we also need to import base64 and we also need to import requests... and we need to define this variable
which is API endpoint and let's define this variable over here, API endpoint, and I'm
just going to copy and paste this value over here... okay... now we have an error over
here because this is going to be... value... X, and this going to be value Y okay and
now everything is just fine and the only thing we're going to do is we're going to copy this
result image and we're going to put it over here and that's going to be pretty much all,
the only thing we need to do because remember here image is a numpy array we are expecting
a numpy array but actually image over here is another type of object because we are opening
this file using pillow so this is another type of object we are going to do something like this...
this is going to be image I think the best way to do it is something like this we're going to
convert it as a numpy array, np asarray, and we're just going to input image and if I'm not mistaken that's
going to be pretty much all and obviously we need to say something like this... no something like
this over here so everything else is just fine so let's see if it works let's see if it works I'm
just going to save the changes and let's go back to my browser, so I am going over here and I'm
going to refresh, I am going to select an image over here you can see we have the buttons we have
this image I'm going to click over here then I'm going to click on remove background and you
can see we are executing the code so we need to wait a couple of minutes... and everything is
working just fine so everything is amazing everything is working just fine, if we click on
original we're going to see the original image the only thing we're going to do now because
obviously we do not want to call this API every time we click on remove background we only want
to call this API the first time and then we are going to let the user to just go back and forth
between original and remove background but we're not going to call the API every time the
user clicks on remove background so this is what we are going to do once we are here I am going to
cv2 imwrite and the filename will be something like this... I'm going to do it first and I'm
going to explain it... in a couple of minutes... okay then this is going to be filename
and then we are going to write this object over here and then we are going
to... I'm going to define filename over here and then we are going to say something like
if os path exists filename the only thing we're going to do is to call cv2 imread filename and
this is very important cv2 color... oh no sorry cv2 image unchanged... sorry imread unchanged
okay because this is an image with four channels so we need to make sure we are reading the
four channels and this is going to be result image... else we are going to execute everything else.... okay so we are going to execute the API
we're going to call the API the first time we click on remove background and then we
are going to save the image we get we are going to save the mask we get we're going to save this
image with four channels into our disk and then the only thing we're going to do every next time the
user clicks on remove background we are going to load the image from our computer and we are just
going to visualize the same image we just saved the previous execution right this way we are just
going to call the API once and then we're just going to visualize it all the times we want this
a very good way to do it and I'm going to import os... and let's see if it works this going to
make the entire process way more efficient, so I'm just going to refresh... I'm going to
load the image again I'm going to click on the image over here I'm going to click on
remove background now we are calling the API we're executing the API this is going to take
a few seconds... we got the response if I click on original we are getting the original image and
if I click on remove background again now we are visualizing the image with its background
removed but we didn't call the API again we are just loading this image from our computer
and now I can just go from original and remove background very quickly without calling the API
so the only thing we need to do now is to change the background because remember this is the web
application we are building so we need to change the background but that's only a very very small
detail we have obviously completed this process and now let's see how we can complete... how we can
change the background, so in order to change the background we are going to use this function
over here so I'm just going to copy and paste this function over here and then I'm going to
call this function and I'm going to input the file we are going to use... which is this image I how
here bg.png so I am just going to input the image like this... bg.png and that's pretty much all let's see
now, I'm going to refresh and we have a very small mistake... I think we need to do something like this... we need to call this only once... okay now I'm going to test the process again
I'm going to click here I'm going to click on remove background... and that's pretty much all
so if I click original this is the original image if I click on remove background this
is the image we got with its background removed so this is going to be pretty
much all for this tutorial my name is felipe I'm a computer vision engineer and
this is exactly how you can create a web application using python streamlit and the segment
anything model, thank you so much modelbit for sponsoring this video and this is going to
be all for today, see you on the next video
