Unlock LoRA Mastery: Easy LoRA Model Creation with ComfyUI - Step-by-Step Tutorial!

hello everyone and welcome back to Dreaming AI my name is nuked and today we are going to learn how to create our very own Lura as I mentioned in a previous video Lor ra stands for a low rank adaptation and is a training technique used to teach large models new things faster and with less memory imagine having to teach a computer to understand human language like you would with a virtual assistant such as Siri or Alexa with Laura instead of starting from scratch every time the model needs to learn something new we retain what it has already learned in the past adding only this new part so when it learns something new it does some more efficiently this also helps the model not to forget what it has already learned furthermore this technique is very intelligent in managing the model's attention helping it focus on important details during learning and finally Laura also makes the computer's memory usage more efficient which means it can learn new things with fewer resources just like learning a new language without having to study for hours and hours honestly I've been wanting to understand how these models are created for a while but I hadn't have the time to study how it all works recently a node has been released that allows us to do this directly from comfyi saving us from having to install alternative interfaces in fact usually we would have had to rely on an interface called coh on which the code of this node is based however if you want once you become familiar with the basics you can comfortably proceed to use Coya directly which surely has more features and parameters for total control over the creation of our model well before we start our workflow we need to create a data set in our example it will consist of a series of manga style images now creating the data set is one of the most important parts and should be done carefully keeping in mind that the image must clearly convey to the model what it should imitate as described by the author of this custom node in a post on Reddit it is important that our data set although varied is of quality and contains material that immediately communicates to the model what it needs to learn since this is just an example I simply searched for some mongy images on Google and downloaded them I won't show them to you just to avoid copyright issues as for the folder structure you'll need to create a general folder for your style or a character you're creating the Laura for in my case I created the manga style folder but the name is only for organizational purposes as it has no value in training inside it you must obligatorily put one or more folders renamed in this this format uh number underscore description the number must be greater than zero while in this case the description can be anything as it will not be considered in other cases such as dream boof training which is not not supported in this node this format is used to identify the number of repetitions of the data set to be processed and the class name which is practically the category describing the images inside but I repeat that for Laura training these two things will not be taken into consideration all right so let's proceed with the installation of the necessary nodes which are image captioning in compy UI and Lura training and compy uh for both I'll be using my forks for now which I have also sent as requests to the original node author in the hope that the changes I've made will be included let's proceed to download them with G clone into the custom nodes folder if you're not sure how to do this you can refer to a video I made in the past on installing custom nodes upon the first launch of comfy UI the necessary dependencies will be installed pay attention to message like this as if they appear you'll need to restart comfy UI after it finishes so that the node works correctly if you're using the original node you'll need to install the dependencies listed uh either in the wi requirements. text file for Windows or the requirements. text file for Linux before starting compy to avoid problems since I'm currently without my Linux operating system I couldn't test my Fork on it so it's very likely that you'll also need to use the requirements stop text file to complete the operation for the ladder great now let's divide our workflow into three parts uh the first part is where we associate a description with each image the second part is where we perform the actual training and the third and final part is where we test our new laora so let's start with the construction of the first part load the war captum load node where will set the folder where our images are located in my case it's this one just a quick note among the changes I made I also introduced support for JPEG files which are still missing in the original now in the original example wd14 tagger is used to tag each image um since I recently introduced a model called Joy tagged in my GPT node which is known to provide better tagging than the models used by wd14 I decided to use that instead however you can use whichever you prefer so let's open the uh GPT saver loader node and connect both it and the Laura caption load to the GPT teex sampa node finally open the Laura caption save node and connect the fields in this way in this last node there's also the prefix field which is simply our keyword that will then use and our prompt to activate our Laura in this case I'll use my manga it's not mandatory but it's recommended to facilitate the use of the model perfect now let's start the workflow as you can see text files containing the associated tags for each image have been generated in the image folder if you want fewer you can modify the max tags parameter of the GPT text sampler now for the boring part open each text file and check for any inconsistent or strange things between the tags in the image for example here I have these tags that are useless as the correct one is only two boys as tedious as this part may be it's extremely important because wrong tags compromise the model's training and also as you can see our prefix is listed as the first tag once done we can proceed to the second part which consists of actually launching the training um to do this I'll use the lower training node in comfy Advanced you can also use the basic one but I prefer using this one to explain some settings in detail scrolling from top to bottom we have ckpt unor name which is the name of the model from which we'll start to create our Laura model V2 is the option to enable If the previous model is a model belonging to version 2.0 of stable diffusion uh which means all those models that have a base image size of of 768 x 768 instead of 512 x 512 and the sdxl models however are still not supported as of now but un confident they will be SIM network module determines the type of Laura Network used impacting the model's architecture and computational characteristic a ladal link in the description if you're interested nexted Precision enables training with mixed Precision to optimize memory usage especially beneficial for gpus with limited memory I will use bf16 since it's supported in the Nvidia RTX 30 series save Precision specifies the Precision with which the model will be saved ensuring compatibility with different Hardware configuration Network Dimension uh defines the rank of Laura influencing the model's expressive capacity and memory requirements rank is the number of simultaneous interactions the model can consider during data processing I know however that increasing it will increase also memory usage and training time con dim specifies the size of a network Matrix called com 2D used for training affecting feature extraction and computational efficiency Network Alpha set the alpha value to prevent underflow and ensure stable training crucial for numerical stability during optimization training resolution determines the resolution of training images is impacting the level of detail captured by the model for SD 1.x the default value is 512 for SD 2.x it's 768 data path specifies the location of the data set folder essential for accessing training data be sure to enter the path to the folder containing the data set folder here not the direct path to the data set um in our case I'll enter the path to the Monga style folder batch size determines the number of data processed simultaneously during training affecting the rim usage and training speed uh with 10 gigs I wasn't able to go beyond two MAX train epox sets the number of epoch for training balancing training duration and model performance for this example we'll train our Network for 400 steps while Laura models that you usually find on CIT AI for example are trained for about 10 20,000 steps and on many more images save every neox specifies how often the training progress is saved T tokens controls the shuffling of tags during training reserving certain tags from shuffling Min SNR gamma specifies the Min SNR waiting strategy influencing the importance of different data samples during training I'll also leave some information about this in the description the next two parameters instead set the learning rate values of the Tex encoder and the unit so the learning rate identifies how fast we want the train model to learn if it's too small we risk the model taking too long or never learning if it's too large it's possible for the model to learn to approximately uh learning rate schedu chooses the learning rate scheduler which dynamically adjusts the learning rate during training to optimize convergence LR restart Cycles determines the number of restarts in the cosine with restart scheduler influencing warning rate adaptation Optimizer type specifies the optimizer Ed for training output name determines the name of the file for the Laura model algorithm is the algorithm used in the network module we select earlier Network drop out controls drop out regularization to prevent overfitting improving the models generalization ability clip skip specifies the recommended layer level for the model selected in ckpt name output dir defines the directory where the L model will be saved by default it's comy ui's directory in the end we have tensor board that enables an interface commonly used during model training to visualize the training progress in the original node it was executable with a separate node but I preferred to integrate here for practical reasons the interface can be viewed at this address once the training has started well after probably the longest explanation I've ever given for a custom node let's proceed to execute the training great now that's done let's try using our Laura like this in this example since our training has been done on very few images and for very few Epoch all exaggerate the value of strength modelon normal in properly trained models in fact this value should be around 7 in the positive clip I'll put the prefix we set earlier for this example I'll also include a flow that shows us what the image would look like without Laura so we can see if we've actually had a significant impact on the result well I'd say that for a model trained with only 400 steps and on very few images it has had a significant impact indeed great and with that we're done for today I hope you enjoyed the video and that it's not too full of inaccuracies since I'm also learning along with you um also I've never been very good with these things but I really wanted to sincerely thank all the people who have decided to support me both on YouTube and on patreon and all those who have decided to subscribe and leave likes believe me when I say that I never thought I would get all this attention just by doing what I love so once again thank you so please consider liking and subscribing if you found this tutorial useful also if you have any questions please let me know in the comments below I'll be happy to help you out as much as I can and until next time keep dreaming

Channel: DreamingAI

Views: 5,073

Keywords: training, lora, train lora, custom node, own model, generate images, ComfyUI, basic, texttoimage, AI, stable diffusion, artificial intellingence, dreamingai, ai news, best free ai, best ai model, dreamingai tutorials

Id: 7rg9O_QRQH0

Length: 14min 41sec (881 seconds)

Published: Sun Mar 17 2024