Connect Your Images to ChatGPT With Embeddings in Azure Cognitive Search

what are you up to I'm searching hard to find a picture of you wearing taco costume and dancing in Halloween that was hilarious of course you cannot find them I deleted all of them I just actually recovered them and I'm gonna show it to our friends [Music] [Applause] all right give me the laptop a few moments later but seriously you remember that we created a video that showed how we can connect our own Text data to chatgpt and start chatting and asking questions about our data can we do this with images or maybe even videos you can actually chat with your images as we chat without text Data through creating embeddings out of your images and retrieve them from your vector database so we can yeah that means I could just say hey give me the picture of mgv airing charcoal custom in Halloween and boom the images you have that image in your database then yes then show it how to do so then let's go hello my friends and welcome to another video which we are going to discuss how this time you can chat and retrieve your images we already previously recorded a video that how you can chat with your own text Data even tabular data through connecting them to GPD models or even any large language model but in this video we're gonna see how we can use Florence API which is a custom Vision a recent custom Vision model on Azure under cognitive services that can help us to understand the context of our images and even support for creating embedding out of the images so when we have embedding out of images what we gotta do we're gonna save them in the vector database and remember recently Azure cognitive search added the support of Hosting your embeddings like a vector database then in this video we're going to use cognitive search as well on Azure to host the embeddings that we're gonna generate through the support of Florence API a custom visual model on Azure to be able to chat and retrieve our images but just type in what is the type of the image you're looking for and get that image back for you from your own images and the database that you're hosting those images then let's check it out before we start make sure you subscribe and hit the Bell icon so you will get notified for the next video alright thank you as we discussed in previous video I already recorded a dedicated video just about Vector database in Azure cognitive search that was a recent announcement that means you can generate your own embeddings using let's say open AI models bring your ward embed into cognitive search on Azure and then later on when you want to chat with your own data or retrieve your own internal knowledge when you have your own chat GPT chatbot as an example then you will do word embedding similarity calculation you have your text as an input as a question you generate embeddings and then you have the embeddings of your own documents so you will calculate the similarity of your question embeddings with your knowledge embeddings to see which chunk of data is relevant to your question to get answered by GPA models right I will definitely watch that video out before watching this video I will add that video to the video description here and also on the top right of this screen make sure you check that out I have already fully explained how we can leverage this technique but as we talk there we use text and our knowledge was text too so I was chatting and asking questions about my own data which was text using word embeddings and if you remember one of the benefits of vector database on Azure cognitive search that we discussed was that embeddings cannot be just based on text you can generate embeddings out of images or out of your videos to videos or just a couple of images right this is exactly what we're gonna discuss about so if I can generate embeddings volume sorry Vector embeddings out of images does that mean I can chat with my images too the answer is yes and this is exactly what we're going to discuss and showcase you a demo how we can do this and the trick here is we're gonna use another cognitive service we're going to use cognitive search as well but for creating embeddings out of image we're going to use Microsoft Lawrence Foundation model this went to public preview I think they announced on March 7 20 23 so that's not really that old and what is Florence model it's a New Foundation model trained of billions of text and images so that's a competitive Vision model and here's the key it is trained with billions of text image pairs this is exactly what we want we want to have our own images let's say I have a company have ton of images right and I have to sort of generate embeddings out of what those images are containing but let's say I have image of a cat and how can I generate embeddings out of the image of a cat so I need to sort of convert those images to based on the text that talk about the content of that image and then generate embeddings so that means I need a model that understands the pair of text to image remix to text sort of that's the whole idea and this is why Florence model will shine which is a a part of azure cognitive service service vision and it has great great actual capabilities to show you a quick demo here they have added this nice video as you can see this is a video given to Florence model so it is sort of describing what's happening there's a cheetah sitting on a hill a woman in in sunglasses and cheetah sitting in glass next next to a car a couple of people in the car so you'll see that it can understand the context of what's going on in an image and here's a video which is multiple frames so frame by frame image by image it can perceive what's going on and you will see that image text pair is showing itself we do have text the context and the given image that means if I have the cheetah picture here and I will know that this picture is talking about cheetah sitting on a hill then I can just simply generate the embeddings out of this and next time when I ask a question my question gonna be generated to embeddings as well then I do similarity calculation and get the image back exactly what we did we chat with your text data so I want to have a chat but let's say to ask hey give me the pictures of cheetah sitting on a hill or on a grass beside the car based on images of my own data set and that's it that chatbot should retrieve all the data back to you so what we're going to use we're going to use Florence model to generate embeddings and stuff then I'm going to use Azure cognitive search to save these word embeddings or text or embeddings coming from these images in a vector database of cognitive search and later on query my cognitive search based on the image that I am asking so let's see how we can do this first of all before I start chatting with our images we have to generate embeddings right and for generating embeddings I should sort of call that Florence Azure model to receive my images and then sort of give me the images right so in order to do so I need to code this and we have coded this process and hosted in an Azure function and this is my Azure function I called it imagemg that does the process that I told you I need this Azure function because when I go to cognitive search to store my embeddings this function gonna generate embeddings for me now let's go to the code that how we create this Azure function how I deploy that and provide you walk through to make sure you understand what's going on in the code so let me open up my vs code before I actually open up my vs code let me show you all my app folder and by the way I I downloaded this code and this app Azure function from the GitHub I will add the reference GitHub to the Discord channel on the reference section so go to the Discord channel the URL of that channel is on the video description go on the references you will see the GitHub report that I downloaded all this code and also the Azure function code I'm going to show you right now so these are typical files that comes with Azure function if you want to create an Azure function app you have to have a file called requirement.txt for example that talks about the pick packages you need to have in place I just opened that I need Azure function and request get installed pip request peep Azure function inside my app to be working let's click on host to see what it is there you go it just opening another tab so it is talking about some configurations that we need to have in place without Azure function for example do we need to have application inside for logging and stuff enabled or not so again these are typical configurations you can use them as is I didn't actually change anything when I grabbed it from the reference and the main important part that we need to have the code that generally embeddings and stuff is actually here get image embeddings and this init dot Pi code is the one that is doing all this magic so let me open up this in the dot point I already have it open actually let me close extra tabs there you go let me also minimize this okay so let's see what this code does in Azure function so it is it is it is going to be an API right so it's going to be listening through HTTP request so it is listening for any call that you want to have that means hey I'm calling you because I have an image I'm going to give to you to give me the embeddings using that Florence API right oh okay so it will receive your input and what's going to be my input my input gonna be the image URL and the record ID so what are these These are the image name and image path that I have them stored in a storage account in a blog I'm gonna show them to you so this is simply just receiving the list of the images URLs and the the path of them in the blob then I call this API and what's going to happen then I'm gonna generate embeddings how are you calling this function I want to generate the embeddings of an image with the given image URL which is sitting in my blob but how that function works that function actually has been defined here so you will see that it is calling my cognitive service endpoint NK I'm gonna tell you where you can grab it I deployed this Azure function and I remove because these are credentials but I can show you how I did and then in the URL section you will see that it is going to under computer vision of this cognitive service and then for I want to vectorize an image so that's going to be the Florence model and then based on the given key it's calling that Florence model and there you go then my embeddings are generated here so I'm retrieve them back as a vector here and that gonna be my output which I'm gonna return as a response body which is coming from values and the values are appended vectors plus the given image path this is your vector and now my Vector is ready so each time that API is listening for generating a Mannix code listening 24 7 like an API that generates image embeddings using Florence model right but before I close this let me show you how I grab these two you need to create a cognitive service resource let me show you on Azure portal there you go as you can see I am under Azure AI Services just type Azure services on the top or cognitive services in your search and you should create one simply and after you create you come here you can see overview activity log stuff and under key and endpoint you have to copy the key and copy the endpoint to add it actually here in the code which I will show you this is the endpoint and this is the key that's it now what's going to happen I have this code that I showed you I'm gonna deploy this to my Azure function right you have to first create an Azure function if you haven't just type Azure function here on the top I have already created one image mg so this one you will create this but no function has been deployed to it yet so you will see here under name nothing but because I have deployed that my function is deployed over this so you're going to create sort of an empty Azure function that is waiting for you to deploy something let's see how we can deploy this so here again we talked about this is my code but how did I deploy this you should navigate to the folder that you have downloaded all those app files or function files from that GitHub URL that I showed you let me actually show you that GitHub URL to make sure that you understand how you need to um copy those codes that's the GitHub URL again I will add that link to the Discord Channel um and you can see the the function the code that we're gonna start which we haven't went through that yet it is vector image and the function the Azure function quoted stuff I talked about is under here functions so you need to sort of download anything on the functions that I call it app in my um code actually here so app is equivalent of this function okay now going back to my vs code again so first thing make sure you're in the directory that you have that app or that function and you need to do AZ login so make sure you log in to your Azure subscription when you hit AC login make sure htcl is installed you hit enter it will ask you to put your email and password and you log in and that's it and after that the only thing that you need to do let me double check there you go you just need to type func Azure function publish name of your Azure function and python so what is that name of azure function remember I told you you need to create one there you go and that's it this will deploy this code or function to generate embeddings in your Azure function and that's how you will see your function has been generated and it is listening so first step a function that generate embeddings is done and deployed in Azure function but I'm gonna index these embeddings in cognitive search remember we're gonna query our stuff index time so how can I sort of call these Azure function from cognitive search to generate embeddings and then host those embeddings as a vector database in cognitive search this is what we're going to discuss about something called a skill set in Azure in a cognitive search so let me go to my cognitive search this is my cognitive search right and that's a search let me go to overview actually so this is the search service that I have and there is something called a skill set what is a skill set skill sets or specific functions that they can be ml based functions that do some transformation on your data when you do indexing for example you have training machine learning model that detect any name of a drug um medicine and stuff inside your given text so you want to call that model when you do indexing your data that hey when you index my data call this model that I have trained to detect the drug names inside the tags and and have them as a field in my index data that means when I search my data I can use the drug name to search those chunk of data so you can bring your own ml functions your function your own Machinery models or some pre-built skill sets that Azure cognitive search give to you so that's very powerful too one of the different type of skill set that you can have is bringing a web API and that VP API is our Azure function API that's the API that generate embeddings so I'm gonna add that Azure function we just deployed as a skill set to cognitive search to index my data by generating embeddings and indexing those embeddings let me show you how the rest we have done that through code and again the code is added in Discord channel so let's see how we can do this without manually doing this so going back to my vs code this is the main code that gonna create all those indexes skill sets and generate the vector database and then query our images before we go through that let me show you the images that I've used again from the same GitHub repo so I do have already a blob of storage I created let me go to my Azure portal uh okay let me find it there you go I just found it so this is the storage icon that I have I have created a container called Vector sandbox and under that you can see I have some images and by the way I have enabled anonymized access so these are not private for this demo so now I have these images what I'm gonna do I'm gonna index them by Florence API embeddings and then query them by chat let's see how we can do this so going back to my vs code this is what we're gonna do Azure cognitive search Vector search code sample with cognitive Service Florence Vision API for images now based on what I have explained so far in 15 minutes you should understand what this title is these are the stuff you need to install for connecting to cognitive search connecting to storage blobs blah blah blah so make sure you have them all in place going all the way down these are coming from the code sample you need to import this that we're going to leverage them in the code and more importantly these gonna be the credentials you need to have for running rest of the code so what they are the first one the cognitive search endpoint that you want to have the embedding sort of index there how to grab that let me show you going back to your cognitive search this is my cognitive search click on keys and you will see the sorry this is let me double check sorry going back to the overview the end point is on the right side on in front of URL and then under Keys we will have the key that you can copy them because when you come to the code you need to have that endpoint and key paste it there and you are going to create an index by all these embeddings right so give it a name I that's my my random name I added image five I was trying with multiple numbers so I made some mistakes that's why the five is there but it doesn't matter now what is cognitive service endpoint that's that AI service that has Florence and stuff we created right so how did I grab that let me show you let me go back to my Azure portal there you go so this is my Azure AI service go under keys and endpoints copy both of them then when you come here put your endpoint and put the key there now remember that we created that a skill set coming from we're going to create that skill set coming from an Azure API function this is exactly my Azure function endpoint so image mg is the name of the Azure function that I deployed I showed you and this is the name of the function that I deployed under this Azure function so here's the endpoint that we have next blob connection is doing that's the place that you have stored your data sources your images so how did I grab this let me show you again how we can do so so you will come to your storage account and then under access Keys you'll see that the connection strings there you can show or copy that by the way these are credentials you can have them in operating system environment in a safer manner grab them from Azure cable right and you just paste it here and the container name the container that you have saved your images I showed you mine is Vector sandbox that's it you're done so I ran this it was pretty quick and then after that to just make sure you have successfully connected to your blob storage this will give you the the first URL of your blob so if I click on that you'll see that this is one of the images that I have saved there so a lady viewing red hat okay let me close this now what are we going to do I'm going to connect my cognitive search to this blob because this is the source of data icon index at right so let me minimize my terminal and by this code I'm connecting to my cognitive search and I'm saying that hey I have a container in a blob storage that I want to create this as a data source in my cognitive search the given name and that's it the data was created so let me show you what's going to happen in my cognitive search when I run this so you will see that inside my cognitive search service data sources I have image for blog I created this one when I was testing the code so we can ignore them but now I have the source of my data from blob Define as a data source in cognitive search that means this code worked properly now creating a skill set remember I told you I'm going to create a skill set to to use that Azure function this is how we can do so so here I'm saying let me maximize this I'm saying that hey I want to create a skill set in Azure cognitive service which is a web API that's going to be my Azure API function and this API gonna receive the image URL which is the storage path and the storage name of that given image and then this is going to Output some vectors which are the embeddings for me so I Define that scale then I add that skill set to my cognitive search and I should have this image for a skill set created well let's check it out I'll go to the skill set their image 4 skill set has been created and if I open that up you will see that oh it actually defined my Azure API function as in a skill set to extract image vectors beautiful now how do we connect this skill set to the index not yet let's create the index first so going back we're going to create the index so here I'm saying that hey I want to create a sort of an index for my images using Azure cognitive search python SDK here and I'm saying that in this index I have this field the ID the image URL the title of the URL and the vectors or embeddings of that URL and it's gonna be a single type this is the dimension this is going to be searchable and because that's a vector I need to do Vector search so I can specify more configurations that how we're going to do Vector search for example what's going to be the metric cosine calcularity or what's going to be the the ranking stuff H and estw that's the abbreviation of the function I told you about so here I have now created the index and I want to add that ink next to my cognitive search that means image for index has been created let's go and check that out so I'm going to back to my cognitive search indexes there you go image 4 has been created but it doesn't have embeddings yet I should connect sort of that skill set to this index to to load and call those embeddings and load them to the next how using indexer so let's create an indexer now let me go back to my code creating an index series here this is dynamic convention added to the sample code and again I'm saying that hey I have the index created but I want to have this indexer which gonna have these fields and this is this image Vector coming in and add it to my cognitive search or updated that means that image 4 indexer has been created so if I go back to indexer you'll see that I I try to multiply as it failed but now this time worked so this is actually the guy that called my Azure API through a skill set Azure function API through a skill set to generate embeddings and have them indexed in the index that we created right that means we are ready to chat and query our images by text but before I do so let me show you the index what's going on with the index we created let's check for example let me search a random number I just want to see the results oh sorry let me go to the field there you go so you'll see the ID image URL the title and the vectors are sort of generated inside this field so if I go back to my now vs code this is the code that what I'm going to do I'm gonna generate embeddings of my question for example I gonna search for a black color hat this is a text as an input but the output is going to be an image based on my images in a blob storage so I'm gonna receive this text I need to generate embeddings of this text too by this function that is calling again the same cognitive service endpoint the Florence API stuff then when I have this query of sort of converted to embedding then I host create this as a vector and I say that hey I want to search my Azure cognitive search embeddings and give me the top one search results from based on the image Vector that I want to do and see how close the embeddings of my question is to the embeddings of my image and then give me the image URL and the title so I search black color hat and there you go it grabbed the image where coming from my data set in my own blob storage without me even knowing let's say this image exists it is my internal data I'm chatting with that so you can even extend this example to create the chatbot to chat with your images as much as you want let's actually change that black to Red I think I have another image in my blob storage I showed you earlier that has a lady with red color hat there you go not only that if you have an image from internet you can even for example say hey that in my internal images which one is the most similar one to the image coming from this Public Image let me show you what is this Public Image I think I have it open uh somewhere let me double check okay there you go this is a Public Image I think it's coming from on a splash.com so I want to ask the same scenario that gave me my own image that is very similar to this one okay because Florence can understand image text pair remember so going back to my vs code not this one this one there you go so I have these Public Image and I'm saying that give me the most similar images that I have the top three and they run this look at that the first one and quite honest it's very similar to this one right what else the second one a lady with a hat again a lady with a hat so you'll see that that the first one was the most closest one obviously I got the title back I got the image URL back so that was a quick and simple demo to tell you that when we talk about embeddings and now we have a vector in the base in cognitive search you can think about Beyond going just your text documents you can think about your images your videos any unstructured data that you can have the embeddings previously it was hard to generate embeddings out of an image but with these models like Florence that they understand text image pair you can get the context of an image what this image is about and convert them to embeddings and host them in cognitive search index there so you can chat with your images too and think about the chat what we created when we chat with our PDF and documents now you can have images in another index so you can have a chatbot that talk with both images and text and think about a ton of other possibilities that you can enable in your project so I hope that this video gave you the motivation understanding this art of possibility a of course that's not like a project as of now I showed you but that was a demo to really show you the other possibility and that was really my purpose and again I add the link of all these codes to the Discord channel on the reference section I hope you enjoyed the video that's all if you want to get something bad and you want to get it as bad as you want to breathe then you will get a big piece of it and don't forget people who rejected you or didn't believe in you they're gonna watch you so give them a magnificent show to like's video take care [Music] [Music]

Channel: MG

Views: 1,054

Keywords: artificial intelligence, machine learning, openai chatbot, openai gpt 3, openai chatbot demo, open ai chat gtp, Azure, Azure Open AI, ChatGPT in Azure, Open AI in Azure Demo, Azure Open AI demo word embeddings, chatgpt advanced guide, Azure Cognitive Search with Open AI, Image embeddings in Azure cognitive search, Azure vector database for images, Images embeddings with GPT, Connect images embeddings to ChatGPT, Images embeddings in Azure Cognitive Search for Connecting Images

Id: HvRCry2BiBg

Length: 29min 44sec (1784 seconds)

Published: Tue Sep 05 2023