Exploring the New Photomaker Project: Another IPAdapter?

hey guys how are you doing today I hope you're having a great day so far in today's video I will be talking about a new project go maker It's relatively new it came out 5 days ago at the time of recording today is the 20th of January and uh the project is really active the last comment was 12 minutes ago and the idea of this project is to take any character or any face and generate an image using that face so as you can see in the demo here you can take a picture of a person then you can generate images using that base image as a template now I would like to say a big thank you to 10cent off for this project and as it is this project is a stand alone project it have Jupiter notebooks as well as a gradi demo on the right here you can see that the black languages is Jupiter notebook and only 0.8 python the G up user zo zoo I hope I'm pronouncing the name right they implemented a compy extension 14 autom maker to confi now it's called as The Unofficial implementation of photomaker it does have a few targets when trying this project you may end up with a couple of errors in fact at the time of of recording this video we have 24 issues but the project is very active and it's being updated every day and we can take a look at a couple of example here where we have a face and we generate a pixel art character or Disney character using the same face and we have a couple of example we have the photography type of uh image coupled down we have Disney characters and the project itself the com UI implementation does provide a couple of Styles and as you can see we can generate it using new style to Disney character digital art line art and so on I'm going to leave links to all the resources I will be using so you can find all the links in the description below the compi extension like I mentioned does give a few errors and I will show you how you can fix those however do know that my system I have a GTX 1650 with only 4 gigs of vram I have 32 gigs of system RAM and I'm using Windows 11 if you're using a different operating system the errors may be different in that case you can leave a comment in the description below and I can take a look at the error to try to help but I cannot guarantee that I will be able to fix all the errors always come back to the project if you are using the confy Y extension go into the issues Tab and check if somebody else got the same issu issue so now I'm going to show you how you can install the confy Y automaker from scratch navigate to the GitHub link you can find the link in the description below now as you can see I am not signed into GitHub so you do not require a GitHub account in order to proceed on click on the code drop down and then where it says https click on the copy URL to clib next open your confy UI installation where you have your run Nvidia GPU db8 F then go into confy UI folder find the folder named custom Lo go inside at the top click on the address bar here and then type in CMD press enter this will open a terminal inside your compy custom node directory then check if you have git install by typing git pressing enter if you see an output like this it means you have git installed most of you should have git installed as you be using confy one but just in case it is your first time installing confy y the easiest way to install get would be to type in Wing get Space installed space- e space-- ID space git dogit the G in git is capital and then press enter this one install git then close out of the terminal go back into the address bar at the top here and type in CMD once again and then again do git and this time you should have git installed now back to the custom nude folder insign the terminal type in get clone and then make sure that you have the URL copy in your clipboard and paste that URL in your terminal then press enter it will tell you claring into Pui Das photomaker and this should be relatively fast as it does not install the requirements just yet it is just copying the folder from GitHub into your system next you want to do CD and just type in confy UI Das maker CD stands for change directory and then we want to go into this folder here press enter now if you're enable to change directory inside the terminal you can close out of the terminal go inside the config y with to make a folder and then at the top the address bar again do CMD this will open you in the correct directory now in here you want to type in two dots beled by a backs slash then again two dot back slash again two dots so we have three pairs of two Dot and three backs slashes next type in Python embedded python dmip install dasr and then you want to do requirements.txt so the first part is really important because we want to invoke the python which is inside the python embedded folder then press enter this will install all the required libraries for this you may get a couple of warnings like these do not worry about them just leave it let the installation complete the installation is completed we'll see successful here then you can close out of this terminal go back into your confy UI Windows pable folder then start your confi either from the terminal or by double clicking on the run CPU or run Nvidia andore GPU go check your jumil to make sure that the Photo Maker was imported correctly if you see around here that says import bail something went wrong so you need to go back and do the previous step make sure that you've inbook the correct python when you were installing the requirements.txt if you miss a couple of dots or backslashes you may end up using your systemwide python to install the requirements the different dependencies were in in the wrong directory otherwise if you see import times successful go into your confy y click on clear okay right click anywhere in the blank canvas click on ADD node and you should have a category called Photo Maker and inside you should have the different nodes now at this stage it is best to test using one of the default workflows that was provided with the comy UI or to make a custom load click on the load button now in the download box go into your confy y Windows portable folder confy y look for custom nodes look for the photomaker custom Noe in my case I have only one folder because I am using a fresh installation of confy UI just so that I do not confuse you with a lot of custom noes in your case you may have custom noes already installed in that case you want to make sure that you're going into the confi folder then look into the workflows folder and then choose the first one that says from Hub this is from the hugging face website choose that Jason on press on open and if you're watching this from the future this eror may not happen in my case while I was recording it the developer updated the GitHub repository in which they change the name of this node to new underscore F maker underscore generator when loading the file it is not finding this old photomaker underscore generator so it's giving me this error now if you tried it and it working it's probably because they updated the workflows fine in my case I'm going to show you how you can create the workflow from scratch I'm going to right click click on ADD nude go into phot and then choose the one that says Bas model loader from Hub this will download a diffuser format of realistic Vision XL so it is important to note that this project will work only for sdxl form as far as I'm aware it is possible that if you're watching this 2 days or 3 days we have an SD 1.5 version but as of right now it can only work for sdxl version it will also require a diffuser format if you're using this node we have another node that says base model loader locally and this one takes in a checkpoint since I'm using a freshly installed confy UI I do not have any checkpoint installed so I'm going to use this base model loader from hug so it's going to download the model directly from hugging face and place it in the correct folder and this node will output a pipe this pipe needs to go into an adapter so click on it drag out you can search for adapter and uh again I'm going to choose the one from Hub but if you've downloaded the file you can select the one that says locally now in case you have a lot of custom nude and you can search for it you can right click add nude photomaker and then select the photomaker adapter loader from Hub the same one this this will automatically download the photomaker model from the tensent art original project now this outputs a model and this model can go into a Lura or it can go directly into the photomaker generation node now for now I'm going to go directly into the photomaker generation node so I'm going to right click add note photomaker and then select the photomaker generation then model into pipe next we have a fill image which we need to fill in so we need to input an image right click add node click on image load image now this image cannot go directly into this spill image it needs to be process so we right click again add node go into photomaker and select the reference image preprocessing node take the input image into the reference image and then this image goes into the new Photo Maker generation node now you can choose a positive and negative form by doing your traditional clip text and code but as you can see this base model is outputting a pipe so the creator also gave us a prop style which outputs a positive and negative FR and it doesn't require the clip black traditionally so take the positive from go into positive negative into negative and this point you want to make sure that you have all of these connected correctly so take a minute pause it and make sure your connection is correct next I'm going to go into excel.com and look for some portrit and you want to look for a portrit that the face is not blocked for example like this it is blocked you also don't want to see these effects on the face you want to have a clear portrait of the face so I found one where the face is clearly shown and it's my spring to on so big thank you to you for providing this Photograph click on the download button and save the file somewhere and your Lo image click on file to upload and then open the file now at this point if your image is too large or is not cro correctly you can always take the image and do an image crop and make sure your character is centered rep and then you can take this drop image into the reference image AG reprocessing now I'm going to try without the crop and the positive front and here it says sci-fi closure B of A man I'm going to change this to a woman this IMG is actually a trigger word and it must always follow a class word in this case the word can be man woman and then following that we need to have IMG I'm going to leave everything as is then this image and drag it out into a save node or a preview node like would usually do then I'm going to click on Q promp looking at the terminal we can see that it is loading the component and it will download the checkpoint place it in the correct folder and we'll just have to give it some time to complete the downare some of you may encounter an error where it says kernel not found if you did not get that error and your image generation was correct do not follow the next step for those who got this error close out of that error go into your compy installation confy UI custom nde the Photo Maker hold up inside there will be a file name as Photo Maker N.Y if you have Visual Studio code open it using visual studio code press control F to get the find window here then click on this little icon to expand it and change it into find and replace at the top in the fine box type in B Lo 16 in the replace section type in Lo 16 without the B and then click on this little icon here that says replace all go to file save or you can press contrl s to save close out of the CMD the terminal and then restart confy go back back into your workflow and click on Q prom once again now if you do not have Visual Studio code you can open the file using notepad that comes built in Windows right click on the photomaker node. py click on open with choose another application and then look for new pad click on always or just once I suggest clicking on just once just in case later on you download visual studio code or any code editor now in here you want to press contr F to get the find window then click on this drop down here to change it into find and replace again do the same thing search for B float 16 and then in the replay section put in float 16 and click on replace all then go into find save close out and restart config y now on this stage we can see that it worked the first SP inish and it is currently generating the image now for me like I mentioned before if I'm not sure if I mention it correctly but I have only four gigs of vram and I'm using a GTX 1650 so this process of generating using sdxl is going to take a long time for me I'm going to pause the recording and we'll come back once the image generation is completed okay I'm back back and it's finally completed I've done a couple of tests and I've collected the result and place it here let's take a look at the first result so this is my input image I have sci-fi closer portrait photo of the woman IMG important wearing the sunglasses in Iron Man suit face slim body I and F grain and then I have the default negative PRP and this is the result I've done two generation just to make sure that uh I'm not getting a good seed and it's pretty close to what we have here one thing to note here is that the photomaker just like IP adapters it's not a face replacement it is not taking the base and replacing it in the output image it's basically just identifying the features learning about it taking those that information and injecting it into the model now the model have an understanding about the features of of the PA and so on basically like the the characteristics of that uh input image then it is generating an image using that information it's uh pretty much like IB adapters work so this is the first batch I've done and then I tried doing it with my face and as you can see it's not close it it does have a couple of the features and a third one I want to do to see if it will be able to identify the styze images instead of just real characters and um have fantasy art of woman in magical forest it's well you can be the judge of of that yourself I'm going to reduce the zoom level and you can see it here so if I zoom out a little bit this is the first one second one and third one here now the images are pretty good because it's using the B sdxl moral of course this one was a realistic Vision but it is as the EXL so all the maor are pretty good in terms of quality and in terms of the face being portrayed into the image um I think this one is the best and this one it did not get a lot of features I was expecting maybe to have line HS but maybe what I should have done is in the style name I should have chosen line out and then it will have done in the same style now this is another difference between IP adapters and photo in IB adapter if you input an image which is lined out even without any PR it will output an output image and most of the time in line out so it does copy this style in this case in photole I don't see that it's copying the style at least not with the three different results I have of course we'll have to experiment some more now for me it's taking a long time just to generate one image and two images take about 30 30 minutes for me yeah so it's uh it's difficult to test and to experiment but over the days I will be testing and posting it on my patreon free blog post you can join it for free by clicking on the join for free button at the top and you will get some pose you will see the pose and the experimentation that I'll be doing with this node and while I was recording I wanted to see if I could make the female character into a manale I noticed that the pr got updated so I decided to do a g and update my project because when I would post this video a lot of people would watch it maybe a day or two later so they will be watching it in the future I thought the project would get updated why not just update it and see if it still work and no did not work instead I got a black image using the same image the same prompt uh same steps and all in the terminal it's telling me that there's a runtime warning invalid value encountered in cast and then it's telling me where it is in the image processor inside the diffusers file so I'm not going to look into this issue right now that's is going to be it for me if you're watching this in the future probably got resolve if you're watching it right away and you encounter the same issue just look into the z z GitHub Ros for confy UI a makeer going to the issues and you can look into the issues and see if somebody else encountered the same issue and whether they were able to fix it or not now come the main question will photomaker replace IP adapters let's say the photomaker project gets stable and it's working for pretty much everyone even then I don't think that vaker is going to replace IB adapters IB adapter has its own news and the fact that can work with SD 1.5 models as well well as sdxl is a plus on top of that we can see that IB adapters can take the style of the particular input image and output is similar style and we'll have to see where to makeer goes from now on and which direction it gets updated and where the developer is going to take it so hopefully people becomes aware of it as of right now the project has only 272 Stars and not a lot of people are talking about it I have not seen any videos about what to make uh right now now of course there's also a time zone difference that will have to take so for me it's morning but for some of you it may be night time it may also be January 19th for you so one day behind especially if you are in the Pacific time zone around Canada you'll be around 12 hours behind me at the time of recording so if you've already release a video and you would like to have a shout out let me know I can put it in the description below especially if you do show whatever type of errors and how to fix those would really like to see those again tell me what your thoughts are on the on the project it's a really interesting project and the fact that zo zo was able to for it to conf you is really great again big thank you big shout out to the project by the way in case you're having difficulty running with your maker directly on your system you can go to hugging face and as of right now there is a photomaker gradio application which is being hosted on hugging face it's running on the hugging face GPU so it's really fast you can upload an image add your prompt choose your style and click on submit and you can of course upload multiple images the more images you upload the better the character consistency will be in the output image now of course do not forget that we have the class word as well as the trigger word the class can be man woman or girl and uh the trigger word has to be IMG MH so that is it that is it for today hope you were able to get it running in case not let me know in the comments down below like I mentioned may not be able to solve all the issues but I can try to guide you in the correct direction and if we do get an update which fixes those errors I will be able to tell you that hey we have an update you can check it out and see if it works let me know what your thoughts are on the project and um let me know what you think will happen whether this F Meer is going to be another IP adapter or it's going to replace IP adapter in your opinion in your testing and with that thank you for watching until the very end I will see you in the next video

Channel: Code Crafters Corner

Views: 3,394

Keywords: Photomaker, ComfyUI, Digital Art, AI Technology, GitHub, Open Source, Face Generation, Artistic AI, Hugging Face, Tech Tutorial, stable diffusion, IPAdapter

Id: an6pXHj1ehQ

Length: 24min 57sec (1497 seconds)

Published: Sat Jan 20 2024