Build an AI 'chat with image' app in 10 minutes | Bubble x OpenAI

all right so I'm going to show you how to interact with open ai's Vision API and probably under 10 minutes is my goal so basically in basis what I've created is this template it's an input output type of situation where it's got all the little elements that you need to start interacting with the API so we have an image uploader an area to define a prompt so you'll tell the model like tell me what you see or extract the text from this image and then we have a little area over here where is going to Output what the model responds with so let's upload an image here I have this image which is has some text on it little action list for myself and here in the prompt I'm saying extract the text from the image only output the extracted text so we're going to run this this is going to call the open AI Vision API and then here we go we can see that it has output what it saw so again this is super simple so let's get into how to actually do this you're going to need a couple of things so first things first if you don't have a bubble account go and create a bubble account you're need to create a new app once you've created an App go over to the basis lab extension you're going to add this and activate it while you're in the bubble editor you'll create your account and then you're going to copy and paste the the template so over here in bubble I'm in an existing app I'm going to activate the basis extension so over here in blocks there is this input output shell so we're going to copy that and we're going to right click and paste it and what that's going to do is it's going to create a new page and again this has all the little elements you need to start acting with the API so next thing we need to do is we need toally get our open aai API keys because we're going to set up an API call so if you don't have an open AI account go ahead and do that and then over here in API Keys you're going to want to create a new secret key when it shows it to you you're going to want to copy that key okay because I think you can only see it once and then also another thing you'll want to do is over on the documentation on open aai I'm go to platform. open.com and here you'll see the Vision section we're going to we're going to grab this code block here in just a second but you can open this page if you want or you can go there when we get to that part so okay everything's designed we basically just need to set up the workflows in the API call so let's set up the API call over here in the plugins and click add plugin we're going to add the API connector if you don't already have it and I'm going to click add another API here you can select import another call from curl and what we can do here is if we go back over to the documentation this quick start code block right here or API call we're going to copy that make sure it's selected on curl going to go back to your bubble app um paste that in and click import what that's going to do is essentially going to preconfig that API call I would name it something like open AI we'll say Vision we want to set use as is actions that's going to allow your API call to show up in your workflow over here when we click the button so this is a post request this is the endpoint whoops this is the endpoint that we are calling and then you can select private on these so you're going to keep this content type application Json for authorization that secret key that you just grabbed from your API Keys over here you're going to want to paste that in right here so we'll go right there and then paste in your API key I'm going to grab mine really quickly cool so I've added mine in and then now we're going to go down here and look at the body of the API call so when we pasted that in basically everything is configured for us but we're going to input some Dynamic values so if we go over back into the design tab a users is going to be able to upload an image they're going to be able to write a prompt to essentially tell the model what they want it to do with the image and we want to be able to set those values into the API call so that open AI knows how to respond and also see the image that the user is uploaded so back in our plug-in tab for the API call we're going to since usually set two um brackets right here like that I'm going to say this is user prompt and in this case I'm not keeping the quotes because we're going to set essentially our value is Json safe which I'll get into here in a second but basically just set it up like this where you have two brackets user prompt if I click down here then you'll see that value pop up over here and then the next thing we're going to do is the same thing for the image URL first things first I actually want to copy this image because in order to initiate the API call we need a value and so we're just going to copy this CU open a has provided us one but I'm going to again set two brackets I'm going to say user image and in this case I am keeping the quotes around it again I'll explain here in just a second so if we click back down here you'll see user image and user prompt you're going to uncheck private I'm going to paste into the user image section I'm going to paste in that that essentially URL to an image that they've provided and then for the user prompt I'm going to add some quotes here so quote quote I'm going to say what is this image and so now we're going to activate or initialize the API call to make sure everything's working so we'll click initialize hopefully this works I think it's working because it takes a little bit perfect so we got return values if you go over here to show raw data you can essentially see what the model responded with so this is an image shows a wooden board walk again we didn't look at the image we just kind of pasted the URL but I'm going to trust it at least in this case that this is what it saw so we'll click save and now our API call is initialized so if we go back over to our design tab now all we really need to do is connect this all up so everything happens on this button click so we're going to click add workflow and we're going to go down to plugins and then let's see which one did I use I think it's new API sorry I set this up before so anyways here we can see those Dynamic values that we set up so the first thing we want to do is okay what is our user prompt going to be so again looking at our design tab here we have this multi-line input and that's the name of it multi-line input a we'll go back to our workflow we're going to set that as our Dynamic value so we'll search multi-line input a value and then we want to say format it as Json save so if you just start start typing formatted as you'll see it formatted as Json save what that's going to do is just because it's sending it as Json to an API it needs to format it in a certain way so that certain characters don't trigger an error so that's what that's doing and then when it comes to our user image this is the image upload or picture uploader so back in our workflow we'll set that to a dynamic value we'll search picture uploader a and it's value so okay now everything is configured but when we click this button it's going to call the API but we need a way to know that it's responded and we need to display that right so it'll work likely once I run this but we won't really know because we can't see anything so we're going to set a state to the page and what a state is is basically just storing like a temp a temporary value um so this page is called input output one we're going to create a new custom State we're just going to call this model response we set this to text that's the type so basically when we click the button it calls the API it's going to set a state and we're going to say the result of step one I believe it's choices um first item message content and basically what I'm doing there is the response is structured in Json and so it's kind of like this sort of tree structure and if we go back to our API call uh if I click manually enter API response this is just going to show what the response looked like and here you can see that it says choices and then it's message and then you can see role and content are sort of nestled under message and so that's how you have to kind of like find the specific value that you're looking for when it comes to Bubble so I'm going to exit out of that back in my workflow okay that's going to set that state so now it saved that value to the page but then we need to display it actually on the page and so if we go over here I have this little text text placeholder that I've set up for you we're going to set a dynamic value and so the name of this page again is input output one which I see right here here's that custom state that we just created and I think that should be it so let's try this hopefully it works so we'll click preview all right so let's see here's my image of some text again we'll say extract the text text from the image and again you can do whatever here but I'm using an image with text oh my god of course so we got an error basically it says there's an invalid quotation mark okay oh I know why so over here in our workflow when I set up the API call the picture uploader what we need to do is we need to add so just deleted it out you're going to add https colon and then you're going to insert Dynamic value which is the picture uploader A's value which there we go the reason for this is when a file is uploaded to Bubble when you try and access URL it doesn't provide the https portion and so I think that's why it through the error so let's try that again all right let's upload um robot let's just do this cute little robot and we'll just say describe what you see we'll run this all right so I think it's working this time because it's taking a moment oh my goodness okay perfect so it described in great detail what this image is so we know everything's working and just like that you've got you know a pretty valuable app to be honest you can set this up for clients and do amazing things whether it's text extraction or you know describing what you see so the other thing I just wanted to show really quickly is a good user experience practice is showing things like this when the model is loading or thinking just so people understand what's going on and so what we're going to do here is we're just going to do some a few custom events basically to turn this on and off so what I'm going to do right here is uncheck this is visible on page load first thing we need to do is we want to show it when the button is clicked because that's when we know the model is thinking so we're going to say element actions toggle and we'll say HTML I think that's loading dots so you're going to toggle that which is going to turn it on because we just turned it off right from there we're also going to trigger a custom event the custom event doesn't exist yet so we need to create it so over here we'll click new workflow or add an event custom create custom event I'm just going to call this hide dots okay and so what happens is when this is triggered we are going to hide that HTML loading dots perfect so then if we go back here we'll trigger okay I guess it linked it so we'll trigger that event okay so let's try this again we'll do our robot perfect we'll say show what or tell us what you see in one sentence click run and again so that showed the dots and then perfect it turned it off when the response came back amazing so just in a few minutes you've set up a very valuable API workflow all with bubble so check it out let me know your thoughts if you have questions and hopefully you create some amazing things

Channel: Matt Neary

Views: 7,149

Keywords:

Id: UZIoSJcssgA

Length: 12min 21sec (741 seconds)

Published: Fri Jan 05 2024