Elevate your AI agents using Voiceflow Custom Functions & OpenAI VIsion API

you've not been using the real power of voice flow custom functions today we're going to be jumping into how to build a custom function using some very basic JavaScript and querying the open ey Vision API to get some external data and process it by the end of this video you'll know what custom functions are and how to build one yourself and as a bonus you'll be able to use the custom function that we built for the open a vision API to query the vision API yourself and use its full processing power to make truly jaw-dropping functionality for your clients hi there my name is Michael gimar and I'm a Voice Low Community expert and you may also know me from our platform flowbridge so let me show you what we're going to be doing we're going to be helping our accountants not to have to go manually and classify receipts and just kind of automate that entire process and put it into air table so that we can process it later through our own custom fronted so this is what that build looked like it's pretty simple we're going to go every over every part of this but let me show you kind of what this looks like in real life so I've got my phone here I'm going to send over a message and say hey as you can see I've already been testing it slightly so we're going to get a response there we go let's now take a picture just through WhatsApp of a receipt that's going to now go into WhatsApp all made possible through Flow Bridge so you can learn more if you click on the card on the top right of your screen and as you see it's converted it all into Json and we've sent that over to air tables and as you can see all that information from the receipt is now into air tables and the extra thing we did here is we've added this category field which for our accountant it's going to make it very easy to visualize and understand which of these individual items in the receipt is actually a business expense or learning budget or something like that and which are personal expenses that the user made on each individual item of the receipt so super powerful let's jump into this and see how we can do this in just four simple Steps step one we're going to be creating a new project on voice Lo step two we'll be making the custom function for the vision API and add these categories that it can select from step three is to make the function to queried off to air tables and process the information from the vision API as step four watch the magic happen so I'm starting from a completely blank project here first of all let's go over to the function section if we click on contents and then go to functions I've already imported the opening I Vision API by flowbridge which again you can find in the description here and then click this import button to actually import I've made a couple of changes so first of all we've removed the query parameter here which is a input variable these input variables are shown in the designer when you put the custom function step into your designer itself so we've removed that because we want to query it from the code itself next part is the actual openi query API you can pause if you want to read the entire thing the important parts to this are the actual categories that I want the AI system to uh be able to use when actually setting these categories for each individual item and I've also set this example Jason how I want the structure to look like when it's coming back from the open ey Vision API I've also ensured in the prompt that I don't want it to return anything else except the Json structure no text and also no no code block syntax just the Json itself so let's go ahead and go over the payload this is the payload that we're sending over to the vision API you can check the vision API documentation to understand the structure exactly so in this case we want the query itself to be in there and we want the image URL to be sent over to the vision API for processing we're making a post request uh We've included some headers which include the opening ey token that we are getting from these input variables itself once we have that config set up we can actually do the request itself but then we want to check if the response is okay if everything goes right it's going to go into this last return function which is going to actually be a success path rather than the error path which we can see up here we want to set that content that we're getting back from the vision API and we're outputting that into output vars and then we're also setting a debug Trace to make sure the response is correct from our testing and then we're sending the actual entire response body as well as a vision message debug Trace so that we can ensure that everything is going well and see what's going on exactly so now that we have the first function ready let's put it into the designer and see what happens we're going to start off by uh putting in this block right over here and we're going to also add a capture step so that we have that information let's map this to a different variable let's call it image URL there we go and let's hook the startup into here so it's going to start by sending this message and then we're going to listen for the actual URL the next step is adding a the custom function that we just built so if we go over to the def section click function and then select the actual function that we just made you can see that here are the input variables that I spoke about in the beginning of that code We'll add some information here and then this is the response that we're getting from the vision API and what we want to do with that we can map it to a variable and then process that further along in the build there we go and we can add the image URL and we're going to map the image URL that we just made from the variables uh into this image URL in the function now as the output let's map that as a new variable that we're going to create Vision API response so that we can use that later on in the build itself now just for debuging purposes so that we can understand what's going on and continue on in our build let's go with this success block and let's output what the vision API is actually returning to us so we can just use that through this new variable that we just made and then if something goes wrong let's just make a little nice oops text so let's try this out let's see what happens if we click on the run we're going to be able to test this out and there we go we're getting a response now rather than doing it through WhatsApp just for debugging purposes let's put in the URL itself to the image and then send that over now the expectation is is that we're going to get an adjon response from the vision API great so it's returning us a zero we did something wrong here so let's debug this a little bit further we can see that the open ey Vision API is actually returning the information we want so so most likely we forgot to do something somewhere let's open up this Vision API step and we can see right here that we did not correctly map over the outputs so if we go ahead and do that now with this Vision API response one we should now actually get a response there we go the response from the vision API so we've built the function the first one we're going to build another one as well and we've learned a little bit of debugging along the way so let's go ahead and go to the next part which is actually processing that information and then sending it over into air tables so let's go ahead and create that function now if we go over to the content area again and then go to functions let's create a new function from scratch this time let's call it the air tables processing function let's remove the S create function and then we get this default structured layout with the input variables that we can get from the input variables over here and a return Trace with some information we're not going to be using that but we are going going to be using the function itself so let's first start by going and creating a new input variable that we're going to need which is going to be the output from the vision API we'll call that expense Json there we go let's create that here as well so that we can actually use it into the code and then from here we're going to want to get the input vs so from rs. input variables input vs is where we get all these input variables from and then we're going to want to get the specific one that we need which is the expense Json one so now that we have that let's make the structure and add some of the variables that we'll need to actually query air tables in this case we're going to have a base ID and a table ID which we going to need to use for the ER tables API call itself inside of the URL we're going to need the base ID there and the table ID right next to it from that point on we're going to actually want to convert the Json string that we're getting from the vision API into an actual JavaScript object so let's do that right here using Json parse on the expense Json which is that string we're getting from V Vision API and then we're going to get that as an object into this expense items array variable so the next part to this we're getting this structure from the vision API but we actually need to map that structure into what air table wants from us so let's go ahead and do that part next and we're going to add a new variable that we're going to call uh expense body and it's going to get this expense items array go for the items within it and then map each individual item into a new structure so we have what we what air table wants from us you can read about that in the documentation on air tables if they want an array uh that go has a fueled object and then within that object they have item price currency purchase location category we have these because that's what we named our columns if you have different columns you'd want to map it out to the column names that you gave it so that's what air table wants from us but what we're getting from the vision API is a little bit different we're getting for example this expense lower dash item and price lower dash item so what we're saying here is go over each individual item that we're getting from that Json object from the vision API and then we're going to map them into what the array structure needs to be for the air tables API so that's what we're doing here and it's going to Output that into this expense body variable the next part to this is going to be to create the config for the fetch statement you can do that within the fetch method itself as well but just to keep things a little bit structured this makes it a little bit easier and from this config we're going to want to have a post request sent to air tables we're going to add the headers for the content type and the authorization which is going to be our API token which should be Bearer and then space the API token itself next we have the body which is going to be the information that we want to send over to air tables it expects to be a string so we're going to Jason stringify the data so that it converts from a JavaScript object into a Json string what the a table API expects from us is that we have this main wrapper called records and then within records we're going to have all of the arrays U of each individual record that we want to add which is going to be this mapping that we're doing over here so we've got this part done let's start sending this data over to the actual API of air table so first of all let's remove this part that we don't need any anymore and let's wrap this entire thing into a try catch statement we're going to do that just for cleaning code purposes so we're going to add this try at the end of it get a catch which is going to be the error itself and if it goes into an error so if we if we throw an exception at any point within this try statement we're going to add a return of the path error and with a trace of debug which sends us a little bit of information of what's going on exactly based on the error that we're throwing this is going to throw an error in of itself because we don't actually have an error path even though we're stating it here so we're going to want to add that right here into the path sections on the right side let's make that error perfect so we have that path that now exists within our function let's continue on and put things into the try area so first thing we're going to want to do is we're going to want to send this information over to the air tables API so we've got this fetch statement that we're making and we're going to wait for the response and that's going to map into this response variable it works a little bit differently than how it works with the normal JavaScript fetch you can view more information on that through the voice flow documentation though it's almost the exact same thing so it's not a very big change so next part is let's check if we're actually getting a correct response so if the HTTP status code is going to uh be an okay if not it's going to throw this error which is going to go into this catch statement and then the message that we want in there is this HTP error with the stages that we're getting from it so it's going to map into the catch statements return the error path and then send us some debug information to continue on debugging what's going on so once we've done that let's grab the fetch response which is going to be this piece of code right here we're mapping the response. Json which is going to convert the string into an object into this response body variable that we're creating once we've got that let's just make sure that the respon body actually contains information so we're checking if the response body has any information or if it's not an object if it if it doesn't have information or it's not an object it's going to throw this error saying that we got an incorrect response from the API so that's the last piece of validation that we want to do here now let's actually create the success path so in the case that everything went correctly we've got an okay response from the air tables API we can now ensure that it goes into the success path path and then do whatever we want from this within the designer itself we do need to create that path itself so let's go ahead and add this path on the right side success and that should be it we now know that everything should be working so now that we have it working let's just optimize it a little bit so that we can reuse this custom function for all of the other projects that we want to use this in so rather than hardcoding these space IDs and table IDs inside of the code itself let's create an input variable so that we can just do that from the designer so let's first start to create this base ID we're going to create a table ID and then we're going to create an API token perfect now that we have that into the input variables on the right we're going to want to add that into the code itself as well so let's go ahead and do API token table ID and base ID now that we have these IDs table ID and base ID variables set up here we could just remove this hardcoded part because we already have those variables within our code that's going to map into this URL that's perfect and then we want to replace the API token that we've hardcoded in here and then add that in as the API token right there perfect so we have the actual input variable set we've mapped in those input variables into the actual code Let's test it out and see how it works so let's go back to the designer we have this Vision API response that we're getting already we're going to want to map that into the custom function that we just created called air table processing function perfect so we now see that we have these input variable mappings on the right side which is the input variables that we've created let's populate them with the information that we had before so we'll put the base ID into the base ID variable the cable ID we'll put in right there and then we're going to have to get the API token and we'll paste it in right there then we have the expense Json which we're getting from the vision API response and mapping that into this Vision API response variable so let's map that over here perfect we have the two paths that we created success and arrow so let's go ahead and respond with something there saying everything was added into air table and then the error let's just say something went wrong which we can then check in the debug traces that we've added perfect everything set up let's test it out let's put in the URL again if you want to get it working with WhatsApp we can do that through flowd more information in the description below but we're just inputting it from here there we go we're getting the vision API response we've got the actual Json response and everything was added to air table let's check that out and there we go it's all into air tables with each individual item and the category that we've quered the open a vision API to do and there we have it our very own 24/7 accountant that's going to be listening to WhatsApp and getting in the receipts and categorizing them of course we've used the system to categorize items on a receipt and put them into an air table but obviously this would be applicable to almost anything like maybe you would want to count inventory for your business and query that into air tables or any other API that you're using for your inventory management possibilities are truly endless here anyways again a huge thanks to voice flow for giving us the opportunity to make this video and showcase it to the community voice flow has truly impacted our agency to make sure that we're able to make these builds rapidly and be highly productive from all senses be it development or from building the flow itself so truly thanks to voice flow for making this possible if you want to know more go into the voice flow Discord where there's a huge community of Builders giving help to each other or check out flow which makes it easy to connect social channels to your voice flow builds or a YouTube channel where we're doing more learnings specific to Voice Low this has been Mike and until next time

Channel: Voiceflow

Views: 1,627

Keywords: Voiceflow, Prototype voice apps, create chatbots, chatbot builder, ai, artificial intelligence, machine learning, webchat, automation, assistants, agents, NLU, GPT, ChatGPT, design, Design chatbots, Production, AI automation agency

Id: sJ5QKxIvcfg

Length: 16min 58sec (1018 seconds)

Published: Wed May 15 2024