Intro to LoRA Models: What, Where, and How with Stable Diffusion

hello everyone today I want to give you a quick introduction to Laura models [Music] together we are going to see what they are how to activate them on stable diffusion and how to use them lure models are fine-tuned models which allow you to generate images based on a particular style a specific character or object and if we look at cbdai which is where you can find most of the models you will see that you have here on top right a filter where you can have a look at different model types and you have checkpoint and you have Laura so if we filter for Laura and I don't know maybe stable diffusion 1.5 you can see that there are different type of Laura models and then on top here you can filter as well Force style concept clothing cars and so on and so forth Laura stands for low rank adaptation and is a cool technique for fine tuning stable diffusion models now there are other training techniques like Dreamboat or text one inversion but they can be computational expensive like Dreamboat for example and the quality of the image generated is not always the best and that's where Lord comes in why because Laura models are way smaller than normal checkpoints normal models and the image quality they produce it's stunning so again let's go into cvd AI we can see like look at this example this is a Laura model you can see type Laura here and you will see here download you just have 145 megabytes but if you look at a normal checkpoint so let's filter for checkpoint okay let's stick this one you will see that you have 5.5 gigabyte so yeah the Laura model is way smaller right so how does it work so when you're training with Laura technique basically Define tuning doesn't happen on the full model but just on a little part of the model which is the cross attention layer and this little part although it's a small part of the of the model it's really important it seems to have a huge impact on the image quality the cross attention layer is the part where the prompt and the image meet what is important to know is that thanks to Laura technique the number of parameters trainable is reduced right and this means that also the GPU requirements are less so that's why you have a lower size model which is even quicker to train the only downside which in reality is not really a downside is that you cannot use Laura ordered by themselves alone you always need to use them in connection with another model like for example stable diffusion 1.5 or maybe better there is a another model you can find that on cv.ai which is called any model actually we can have a look at this together so if we search any any Laura any Laura checkpoint which is a specific model trained to make sure that any Laura will work with that I'm going to show you the difference between using stable diffusion 1.5 and this any Laura checkpoint and you will see probably a huge difference but as usual this is very subjective so it's up to you what you want to use and what you prefer so how to activate Laura into stable diffusion in the past well not really in the past but like last month or two months ago we had to activate Laura going in the extension Tab and activating like installing the extension but as of today luckily we don't have to do that anymore and you should find Laura already in your stable diffusion when you initialize it which is amazing right so in reality we don't have to do anything but downloading the models so where you can find these blur models you can find them on hagging face if you go in here you have a long list of Laura models trained fine-tuned by other people or Civic Ai and something that I really like of Civic AI is that I just picked one Laura model which is Studio Ghibli style which I really like and if you go on this eye at the bottom right of the picture you see all of the settings used for getting this picture here so we're gonna test this together and see if it actually works and something that you should know as well here you have important settings so you you have the details of this model and it's telling you the type which is Laura is telling you when this was uploaded it's very important if you're looking for the latest version of the model the trigger word this is super important because this is the word you need to use in your prompt and if you don't use this word in this case it's Ghibli style you're not gonna have the effect you would like to have from the more that you cannot exploit the model well in addition to that we need to also recall the Laura model inside the prompter and we'll see together how to do that so to download the Laura models it's quite easy you just press under the load button and as well when you go into again phase you just need to press on the download button for the Laura model which usually is a safe tensor the download should be pretty quick because the Lora model is very small and once you download it it should go into your downloads folder so what we have to do now we need to just move or copy and paste or cut and paste the save tensor we just uploaded into our stable diffusion web UI folder let's have a look which folder we need to use for the Laura model so this is my stable diffusion web UI you need to go inside models and then here you will see you have different folders and you have Laura you click here and here let me clean it a little bit because it's Horrible's cleanup by name and here we need to move this file inside the Lora folder so I'm just gonna do that that's it I can close my download folder and here you go so just to have a look again where it is like this so you go into your stable diffusion web UI models Laura and inside here you need to have all of your Laura models pay attention because usually you put your checkpoints your full models inside stable diffusion which is in here but if you put your Laura model inside here your lower model is not gonna work so make sure that you paste that inside your Laura folder yeah and if you're using Google drive if you're using the drive an external drive make sure you're using exactly the same folder the path to that folder is exactly the same so stable diffusion web UI models Laura easy okay let's go into stable diffusion now on top here on top left you can see all of the models available these are not Lora models here you won't find Lora model but just the full model like for example stable diffusion version 1.5 which is in my case this one but given I want to use a Lora model I'm going to use this any Laura checkpoint I was showing you before because usually works better with Laura models it's made exactly for that so I'm going to use this one and where we can pick our Laura models so you see here you have this little button under the generate button you can click on it and then a tab like different tabs will appear so you have textual inversion hyper Network checkpoint and Laura where you can see all of the model you have now I don't have textual inversion or other network so I'm going directly into Laura and inside this tab you will see all of your Laura models now the one I want to use is this Ghibli style offset so what you can do you can just click on it and then what happens is that here will appear a small text which is Laura Ghibli style offset one in your prompt positive prompt now every time you want to use a Lora model you need to write this piece of text and in addition to that you will need to write the the word specified in the model description in the case of the Ghibli style the word was Ghibli style yeah so I'm going to copy this and I can go back onto my stable diffusion and I'm going to paste it like this and then I'm going to write my prompt etc etc now what I want to tell you about this this text so this format of this has to be exactly this one it doesn't matter where you put it but it has to be exactly this one so you have Laura column and then you have the name of the model which has to be exactly the same as the name of the model you downloaded if we go into our Laura folder inside stable diffusion web UI you will see that this is my Laura model and then part of the name you need to get is exactly this one so Ghibli style Offset you don't have to add dot Safe 10 Source you just need to copy and paste this part and then you have this one here there are different name for defining these it's way it's a multiplier Alpha this is a number they usually go from zero to one sometimes you you can find 1.7 1.1 or something like that what is it so if you study to zero it means we are not using your Laura model basically you're not using it but you're just using the you know this model here you picked if you give you one it means you're giving full weight to your Laura model so usually when you go into saved AI they tell you which is the minimum weight you should be using for that specific style and then you can play with that so for example in this Ghibli style they tell you this is good around one wait for the offset version and 0.6 Define for the old one so you see they are telling you which weight you should be using which is great now you have three different ways of writing this piece of text you don't have to remember it so first you can write it by doing what I just did so clicking on this button and then picking the Laura model you want to use let me close this now otherwise what you can do you can go inside a Civic AI model details and then I don't know for example if you want to reproduce this picture here you see you have this eye I showed you before you can copy generation data or you can also like if you look at in here you have the what you should be writing in your prompt in here this is your positive prompt and this is your negative prompt and these are other settings but if you copy generation data should give you all of these actings not just those and then you go back into your stable diffusion and you paste what you just copied in here in your positive prompt and then you can click this little button below the generate button and like magic all of these settings will apply to your stable diffusion and you can see here you have this name to recall dilora model and you have this Ghibli style trigger word which is really important just to show you like obviously here you have different example of pictures and if you click on all of them you will see that this Ghibli style it's recurring in all of them because as I said before this is the trigger word so you actually have to use that for reproducing this particular style so ideally when you do this it should work sometimes can happen that they upgrade or they modify something in the model and they change the name of the file but they don't change it in here so sometimes when you do this copy generation data and you paste it in here it can be that this is not exactly as the model name you have downloaded so in that case you need like if something doesn't work if you said if if it gives you an error right you just need to double check that then file name is exactly the same as the one you downloaded so this was the second option and the third option is writing it manually also in that case we are human and we can make mistakes so it can be that sometimes we write the wrong name so just make sure it's exactly the same and also that there are no spaces between words so in this case you see you have 1.07 as weight although one should be the maximum ideally anyway I'm using this ghiblit style Laura I'm using the Annie Laura model checkpoint I copied and pasted the settings from civit AI to here for this picture here so when I'm going to press the generate button ideally I'm expecting to get something similar to this picture let's see if this is the case so we go here we press generate oh it's giving me an error so it says could not find obscaler name r e s r gon anime video that's fine I don't need to use the high resolution for now so I can remove it so it's something related to upscaling the the picture right so I I can antique this high resolution here and I'm just going to press generate again now it's working and well like magic we got you see we got this this picture which is uh probably is not exactly the same as the one we have in here but really similar right so that's how you use the Laura model now let's have a look if we change to I just want to show you like the difference between using this model which is uh any Laura checkpoint compared to using stable diffusion 1.5 which should work anyway because it's the stable diffusion model use for training this is Studio Ghibli let's press generate again let's see what happens yeah and you see that you don't get the same effect at all so what is this so let's have a look yeah definitely not I would suggest you to to try different models if you wish but in my case I'm going to use this any lower attack point for this tutorial now ideally what we can do to use this style on something else like if you don't want to produce this uh picture right but you want to generate something different but still applying the same Studio Ghibli style we can maybe I don't know change something here maybe I want to remove all of this I don't want a person maybe let's try with an animal I don't know if this is gonna happen but let's try so we can maybe generate a cat sitting in the forest and then I don't know we can put something some other additional feature just to Upper detail the okay let's uh let's keep this yeah only just to see if something happens okay and now I didn't give many information to my stable diffusion but as you can see the Studio Ghibli style has been applied which is amazing and instead of having the woman and now I have this little black cat which is really cute even though he has three Tails apparently but yes in this case we need to work maybe a little bit more on the prompt both positive and negative prompt and regarding this the other settings I'm quite happy with with that something very important I think is the seed as well but in general it's more important when you want to recreate exactly the same picture like make sure that the CD is exactly the the same right and in this case the seed is this one I can use a random seed let's see what happens maybe something will change let's press generate again but hopefully we'll still get the same Studio Ghibli Style and we indeed do and probably this is even nicer because it seems more realistic cat and it's it's quite nice so something very cool you can do as well you can combine different Laura Styles and this is when you want to you know have a merge between a different styles or if you want a Ghibli style but mixed maybe with some person some character right so for it let's let's have a look at an example let's have a look at some celebrity or maybe we can use the most downloaded or highest rated maybe it's better so we get a good one you can use this lady here Alexandra Daddario so I can download this model right I'm gonna move the model inside my stable diffusion again folder I'm using Windows now because stable diffusion on my Mac was really slow and color but wasn't working properly today so I had some issues I'm using Windows so you will see a little bit of a different background but the you know the steps you need to follow are exactly the same and I'm going to restart my stable diffusion so I have the terminal here and I'm going to quit this and I'm going to restart it again okay I restarted it I'm going to refresh the page okay now if we go into this button again we should see the new model I just downloaded let's go a little bit down both the new Laura models so I'm gonna press on both of them so this should you know the text should appear here and then we need to add you know the trigger word which in the case of the Ghibli style is Ghibli Style and in the case of this lady I'm not sure which one it is let's have a look so it's Alexandra and then this is the trigger word so Alexandra I'm Gonna Take It going unstable diffusion I'm going to add this in here so I don't forget and then you can start building your prompt ideally and I'm saying ideally the sum of the weight of the two Laura models should be equal to one so if you give 0.8 to 1 it should be 0.2 on the other side and if you give 0.6 you should be 0.4 and the other way around so obviously it works even if you use one and one or it just means you're giving the same weight to both if you use zero for one it means that you're not using the one with the zero but you're using the other one so yeah like just keep this in mind let's for example put I don't know choose 0.841 or 0.5 and 0.5 for both which in reality is exactly the same of using one and one is just for giving you an example where the sum of them should be equal to one this is obviously like it will need more from description and also negative brand it's just for showing you how it works and then you press generate and you should get a mix of this uh of these two right so this is what we get let me open it so yeah it's kind of like this lady right and and The Studio Ghibli style is here which is great and then I don't know if you're not happy with that but you would like to have something still more looking like her you can increase the weight of the Alexandra Daddario weight so here I don't know you can put 0.80.9 for example in here you can use 0.2 in this case and then ideally you should get something similar obviously you can use more than two Laura mod that you can use three four five how many you want you can mix them play with them do whatever you want until you get what you actually want and need something interesting we can have a look at later on it's maybe how to train your own Laura model there is Koya which is a very famous and very quick and simple to use so maybe we can do that yeah I hope you enjoyed and see you at the next video bye

Channel: Laura Carnevali

Views: 128,177

Keywords: stable diffusion, stable diffusion v2, diffusion, ai art, diffusion model, generative ai, generative art, openai dalle, stability ai, ai artist, imagen, nft, apple silicon, m1, stable diffusion on m1, stable diffusion with python, stable diffusion hugging face, stable diffusion github, stable diffusion v1.5, stable diffusion tutorial, lora, lora models, stable diffusion lora, civitai, hugging face, civitai lora, hf lora, use lora, lora sd, image to cartoon, studio ghible

Id: ZHVdNeHZPdc

Length: 21min 0sec (1260 seconds)

Published: Tue May 09 2023