How to serve your ComfyUI model behind an API endpoint

hi I'm Philip from Bas 10 and today I'm going to show you how you can deploy your comfy UI project behind an API endpoint so comfy UI let me bring it into full screen here is this really cool tool for building image generation pipelines locally on your own computer with models like stable diffusion control net that let you build some pretty cool stuff and what I'm going to do here in this video is talk about how you can get these projects off of your computer and into a hosted API endpoint so that you can share them with your friends or build applications on top of them so this video is going to assume that you are already a comy UI user I'm not really going to talk through how to build this here but it's going to assume that you don't really know anything about deploying these models and we're going to walk through all of that and by the way I am going to show a whole lot of code in this video a couple hundred lines don't worry about any of that it's all going to be linked Below in a blog post post and in some GitHub repositories so you don't have to worry about pausing and you know typing it out it's it's all linked down below so what I have here is a pretty standard simple example project with comfy UI that's going to use sdxl and it's going to use control net to basically take a logo and create an image from it um that image is going to have the the logo embedded in it and what I want to do here is take this project and deploy it behind an API endpoint so I can use it in my app so what I'm going to do is to start off I'm going to take a look at um oop sorry about that um I'm going to take a look at what I have in my comfy UI repository so I have that repository cloned locally and then I have these models right where I've downloaded this sdxl model I've downloaded this diffusion model that's going to come in handy later so in order to actually export my project so that I can use it as an API I'm going to have to save it in this API format if you don't see this option on your comfy UI uh don't worry just click on this gear icon up here click enable Dev mode options make sure that's selected and then you're going to have it um so when I click this I'm going to get a file it's going to be this big Json file you know 100 lines or so and this is going to be the file that my entire API is going to be built on so what I'm going to do is I'm going to take this file and I'm going to bring it over to a truss so where did I get this trust from what is truss great questions awesome questions glad you asked so truss is a model packaging and deployment framework it's open source um we've been working on it here at base 10 for about a year and a half now um and it's you know kind of the underpinning of the whole platform and we built this sort of trust template um that's going to to let you take your comfy UI project put it into this template and run it as an API endpoint so we're going to you know clone this template from this trust examples again to actually you know follow along with all the code and stuff grab the blog post from the description that's going to give you what you need to copy and paste into your terminal and we're going to add a couple things to it so looking at this you know cloned trust what we're going to do is first off you know we've got this config doyen it's got all of our python requirements it's also a little bit of infrastructure as code it's going to specify what resources we need and then we have this model.py that actually implements the model server you know loads up comfy UI and runs inference with some you know helper code in here to make it more readable so what we have to supply to this template is two things number one we're going to need to bring in this comfyi workflow. Json file um that we out that we export from from our comfy UI application and just got to make sure that it's in this data folder and it's named exactly like this so the template does come with this exact file in it if you have your own that you're trying to deploy just go ahead and delete this and replace it with yours the second thing we're going to need is this model. Json file and what this does is it you know basically replicates what we had in if you look over here in comfy UI running locally we have these two models downloaded locally and stored in specific folders well all we're going to do is we're going to provide a URL in this case to hugging face where the online server can download this um as well as a path of where it would be stored locally uh within comy UI so again if you're providing your own project make sure that you're using the same models and that you follow this system uh with setting up your model. Json file in this data folder so once you got all that set up we're going to want to push so for that um we're going to go down to the terminal here and we're going to run trust push um this is going to take our packaged model and deploy it to base 10 um if you don't have a Bas 10 account you're going to have to make one real quick provide an API key takes like 2 minutes and we provide you plenty of free credits to get started um if you are experimenting with this so much that you run out of free credits just you know send me an email I'll get you some more anyway so once we get this deployed and I am going to cheat just a little bit here um I've got this deploy set up already it takes just a couple minutes but I don't want you have to sit and wa sit through it in the video um it's going to be running on a T4 instance and um that's like our smallest GPU that we have on the platform you can run this on an A10 a100 uh if you want to get faster results or do some batch inference um but for now T4 is great because it's cheap um and with this up and running the the last thing to do is just invoke the API endpoint to call the model um I got these values through this call model model um created a endpoint URL created an API key with this button here by the way I did uh revoke this API Key by the time you're seeing this so if you're trying to get in my base 10 account don't bother besides it's just a demo account nothing to see there anyway anyway so I'm going to go ahead and run this script to invoke our model um and when I go ahead and run this you're going to be able to see in the model logs the actual influence logs as this gets going so it's going to load the model load the two models and actually run through this process um it does take you know uh a little bit a few seconds 20 30 seconds to um make this output um because it's running on this T4 GPU and it's running this whole comfy UI overo in the back end um so again if you uh if you need faster results we can just bump up to a bigger GPU and in terms of the output that you can expect from this model uh once it finishes up here we go um we've got this uh this Forest top down image and the river is going to be in the shape of the base 10 logo um just like in our example project um in when when we were running locally and yeah we can see here that we're you know using this as the mask for the control net so yeah so this is how you deploy your comfy UI project as an API endpoint um it's a pretty straightforward task using the template that we provided in the trust examples repository be sure to check out in the description the blog post and GitHub repository that's going to get you started and if you're a comfy UI user you can take your projects now and deploy them on B10 thank you so much for watching and please feel free to reach out to me uh philip. KY b.co if you have any questions about this uh this stuff in this blog post

Channel: Baseten

Views: 6,429

Keywords: baseten, base ten, serverless gpu, ai native, fine tuning, foundation models, LLM, generative ai, gpus, kubernetes, llmops, LLMs, machine learning deployment, ml model deployment, ml pipeline, mlops, model deployment, model infrastructure, ml infrastructure, model ops, model pipelines, open source models, serverless architecture, serverless framework

Id: nNhw_AxW7_E

Length: 8min 18sec (498 seconds)

Published: Fri Dec 08 2023