Upscale your Images using DEEP SUPER RESOLUTION with ESRGAN

so you've gone for a leisurely walk along the beach and you've taken some absolutely amazing shots but then you realize you've gone and set it to a ridiculously low resolution and all your images are coming out blurry what do you do well in this video we're going to take a look at how to fix that exactly we're going to take low resolution images and convert them into high resolution images using a pre-trained deep learning model ready to do it let's get to it [Music] so we're going to keep this one pretty lightweight from a technical standpoint so if you are a beginner this one should be a little bit easier for you to get your head around now in order to do this we are going to be leveraging a pre-trained gan model from github link should be up right about now now in terms of leveraging this model there's going to be a couple of key things that we do still need to set up first up we need to clone down the github repository and bring it down to our desktop we've then got to install a couple of dependencies but then we'll be able to go on ahead and test it out with our own custom images so you can take your own images pass it through this pipeline and have high resolution images generated from it ready to do it let's jump into it so in order to give this tutorial a little bit of context we're going to go and have a chat to our client and explain how it is that this model works and does what it does let's go and have a chat yo nick i need to enhance some of my images you got any ai that can do that sure have you heard of esr again nope tell me more well the model can take in a low resolution image and generate a high resolution equivalent using deep learning this means that you can take all your sucky beach photos and make them look great interesting but how does that work it's built using an underlying architecture called a generative adversarial neural network or gan for short so we have two neural networks one generator and a discriminator the generator tries to generate a high resolution image and the discriminator tries to pick it out and determine whether or not it's real or not i don't quite get it imagine you've got a counterfeiter right he wants to create an image that looks identical to the real thing but obviously it's fake so he goes and creates this image and he takes it on over to a pawn shop to try to get a little bit of money for it the store owner then tries to critique that artwork to determine whether or not it's real this is exactly how again works the counterfeiter in this case is the generator and the critic is the discriminator we feed in low resolution images to the generator so that it creates a high resolution image or artwork and then our discriminator tries to catch it out hmm okay so how does the training work training is a bit of a balancing act now historically gans have been notoriously hard to go and train for the esr gan model our generator is rewarded for generating images which fool the discriminator so in this particular case generating high resolution images which look almost identical if not completely identical to the high resolution equivalent of our training images the discriminator in this particular case is rewarded for catching out those fake images so here high resolution images that aren't that good cool yeah awesome let's see it let's do it welcome on down to the breakdown board so in terms of starting out with this the first thing that we're going to need to do is grab some low-res images now let's say we've got some images of our beach scenes how do we draw a beach so let's uh add some sand so see what sand like yellow add a few waves and add the sun in the sky right cool so that is our low resolution image so we're going to take that low resolution image and pass it in to our generator neural network so that will effectively output a high resolution image so this is what we're going to get as our output which is pretty cool because we can take something which is a relatively small image or say a blurry image and bump that up significantly now there have been a number of iterations for this specific model so i actually tried to build one from scratch called s again so esr gen stands for enhanced super resolution generative adversarial neural network now there definitely are other instances of this but the nice thing about being able to leverage the one that we're going to show in this video is the fact that it is pre-trained these are ridiculously hard to train from scratch they require a ton of data a ton of monitoring and they can blow out really really easily so the fact that we can use this and it's all open source is absolutely amazing so we're going to take our low resolution image we're going to pass it through this network and we're going to get this as our output now you're probably thinking well how do we actually train this let's say if we wanted to well first things first you're going to need a bunch of data right and that data comes in the format of low resolution images and high resolution images now typically what you do is you'd get a high resolution image so let's say this is high res add some mountains so you'd actually take up that high resolution image and you'd actually make it smaller so you'd actually convert that high resolution image to be a low res image you then take the low resolution image and pass it through a generator neural network and learn to generate that high resolution image and you'll calculate the distance between the real high-res image and the generated high-res image we then take the generated high-res image which you can see here let's change the color of our arrow and this will get passed through to our discriminator neural network you can tell my discriminator drawing is getting significantly better so that will be a discriminator now that discriminator is also a neural network the thing is that that discriminated neural network is going to take your output sizes so whatever your output size is that is going to be the input size into the discriminator so this is actually taking larger images and outputting a number either one or a zero so a real image or whether or not it thinks it's a real image or whether or not it thinks it is false and that in a nutshell is how the esr gan model works there are a whole bunch of other metrics in terms of the implementation and specifics in terms of the implementation but at the crux of it that is how that works let's jump on over to the tutorial and get this thing going alrighty so let's get right into it now i've prepared a couple of steps that we need to go through in order to get this working but namely the first thing that we need to do is clone down the code so this is going to allow us to leverage the pre-trained esr gann model we're then actually going to download the model itself so i'll show you which one to pick we then need to install a bunch of dependencies and then we'll be able to get through to the good bit but first things first credit where credit's due this model that has been made open source on github is available by zintow so he's a researcher he or she is a researcher at the 10 cent arc lab which is the applied research center absolutely ridiculously amazing organization when it comes to deep learning and machine learning and advancements in ai now the esr gan model that we're actually going to be testing out is amazing these are some example results so you can see there that is up sampling and producing ridiculously sharp images that perform or that look absolutely great so we're going to be able to use this now first things first let's actually get right into it so we are going to clone this down and then we'll go and get that model so let's go and clone this so i'm just going to open up a command prompt and i'm going to go into the folder that i want to clone it into so i want to go into my youtube actually let's open up a full command prompt um so we are going to clone it into here so we're going to type git clone and then the link that i had there so it'll be git clone https colon forward slash forward slash github.com tao forward slash esr again so if we clone that that should start cloning it down and assuming you have cloned it into the folder that you want to so i've cloned it into a folder called where is it yes again let me zoom in on that so you can see that that is cloning over there right now that is the folder so let's wait for that to finish my net always seems to want to go slow whenever i'm recording a youtube video all right cool we're done so if we actually go and open that up now you can see we've got all of our good stuff so we've got our models and right now that's got nothing in there so we actually need to dump in the trained model into there and then we'll actually be able to test out our model by dropping in images into this lr folder but before we do that let's go on to step two so the next thing that we need to do is go and grab the model or the appropriate model from this particular google drive folder now that specific link is from the original github repo so let me show you where to get that just in case it does change so if we actually scroll on down you can see down here there's actually links to download the pre-trained weights so they're available from google drive or baidu drive and if this does change then it will be available in the xin tower github repo so just be mindful of that cool so what are we going to go ahead and do so we're going to go to that google drive link so that's the one that i've got there and you should be able to download the model now there's four models in here the one that i've been using is this one over here so rrdb underscore esr again underscore x4 so if you go on ahead and download that that should be i think it's around 60ish megs yeah so 63.8 meg so we'll let that download you can see it's downloading down there in the bottom corner cool cool so this is going to be pretty light in terms of coding today we're just going to be testing this model out and seeing some of the results that we can generate which is pretty cool we'll give that a sec maybe i'll fast forward this in editing who knows boo yeah that is now downloaded so you can see it's now completed so we can then go on ahead and grab that model so it should be in download so i'm going to cut that and then you need to place this so if this is my root folder here right so this is where i've gone and cloned down the model so you can see it says esr again this is exactly what you're getting from github you just need to place that model that we just downloaded into the models folder so i'm just going to paste that there so effectively you should have the github repo inside models you've now got this rrdb underscore esrgan underscore x4.path model cool so that is that all set up so let's quickly double check our to-do list so we've now cloned our github repo so that's done we've downloaded our dependencies or downloaded our model we now need to install our dependencies so there's three key dependencies that i noted that i needed for this so those were pytorch and ideally you want cuda for this so if you haven't seen how to install cuda jump on over to one of my tensorflow videos i'll show you how to set that up so we're going to need pytorch and specifically a cuda implementation if you don't have a gpu you should be able to try this out might just be a little bit slower and then we're going to need opencv python so opencv 2 and glob2 so let's go ahead and handle this first one first so i've got a virtual environment set up here so i'm just going to activate that if you don't that is perfectly fine what's it called it's called super res scripts activate so if you don't have a virtual environment that's fine you can skip activating that you don't need to go ahead and do that but now that we've gone and activated it the first thing that we need to do is go on ahead and install pytorch so let's jump on over to the pie torch website by torch.org and then from there you can just hit install boom and then you've got this sort of wizard that you can go on ahead and click so we are going to choose the uh stable version my head's not currently covering that stable windows i'm going to use pip python and then choose a cuda 11.3 if you don't have a gpu that's fine just hit cpu i believe you should be good to go i haven't tested it out so if you do get any errors let me know in the comments but for now stable windows pip python cp oh no cuda 11.3 so we're going to copy this command jump back over to our command prompt and i'm going to paste that in so you can see that that is already installed so if you're installing it for the first time might take a little while but this looks like it's saying requirement is already satisfied requirement is already satisfied requirement is already satisfied so we should be good to go okay what were our other true dependencies so we need to install opencv python and glob2 so we can actually just copy this command here so pip install opencv dash python and glob2 paste that into our command prompt so let's clear this make it a little bit clearer paste that in there cool so that's installed already so you can see no issues there but the full command is pip install opencv python and then glob2 so let's quickly take a look at our to-do list now so we've done that we've done that we've done that can hit done all right now it's time to get this thing going so the fourth thing is really all to do with testing this out and it's kind of crazy how straightforward this is to get it running so if we go into our so we need to place our low resolution images into this lr folder which you can see there right and right now we've got two images in there and then we can run python test.pi and that is going to output our results into the results folder cool makes sense all right let's try it so i'm going to bring this over to here and right now we've got uh we've got the test image so these are the sample images if you see this image this is a really common one that you'll see referenced inside of a lot of super resolution gann papers when i was trying to build sr again i actually picked that one up as well okay but what we want to do is we now want to run that command so first things first we need to go into the esr gan folder which we went and cloned from github so you can see that we are now in that cool then we can just run the test command so in here you can see that we've got this file called test right so you can see that that's there so we just need to run that so i'm just going to zoom out and boom boom boom all right cool let's run this so we can run python test dot pi so you can see it's running right now it's a model path so it's picked up our model it's testing the first file which is a baboon and the second file which is a comic now those results are now available in this results file pretty sick right look at the resolution on this bad boy significantly better right so this is significantly larger if we take a look at the other image where was it uh low res that is how small it was how crazy is that even if i zoom out we've gone from that tiny image to this how amazing is that in seriously like virtually no time uh the first time i did this i was absolutely blown away i thought that this was ridiculously sick um let's go and grab some other images so we can delete stuff that's inside of lr let's go and find um some images i'm thinking uh what's the new mercedes uh what is it w13 so this is the new f1 car let's grab this bad boy that's already huge let's grab a small image so uh size let's go to medium all right so uh that's still big no big trying to find images that are a bit smaller all right let's copy this one so i'm just going to save this image and i'm going to save it into that lr folder so if we go into youtube esr again esr again lr so let's save it into there and then let's run that command again so right now in that folder we've got the image of louis's car of course yeah that's the list and then we can go and run this so python test.pi so you can see it's picked up that file so 526 44766 that's that file there boom look at that how cool is that so if we go and take a look at our low res look at the image different like i mean this is up sampled i mean there's a not slight little bit of graining there but it has still gone and upsampled this with a ridiculous amount of precision i think that this is absolutely phenomenal that's actually how big the image is that's without zooming nuts right how good is that so easy let's go and grab some more images and i'm consistently fascinated by this um i'm trying to think uh so sydney harbour bridge small that's that's i've typed in small it's returned an image it's just 6244 pixels by 312 pixels yet nice one okay good ridiculous all right that one's small let's copy this one so i'm gonna save the this sydney harbour bridge image what is this this is 555 pixels by 370 let's actually take a look at the size of the up-sampled luis image so how big is this bad boy now this is now 2800 by 1008 let me zoom in so you can see that so that image of lewis's car is now 2800 by 1008 crazy what did we start out with so if we're going to lo res so it started off with 700 by 467 so how crazy is that so what is that improvement i think it's four times bigger right that would explain why it's called x4 so we've gone from what is this properties so we've gone from 700 by 467 to 2800 yeah so four times bigger crazy how good that is all right let's try a harbour bridge model so i'm going to delete the lewis car and again if we just run python test.pi that's picked up our sydney harbour bridge image should get a result let's go into our results boom how nuts is that i am just amazed by how good this is so that's that's our upsampled image so if we go and compare to what we had previously all right so it's a little bit difficult to tell there because i've made them both the same size but this is significantly larger like we can go way bigger crazy how amazing that is let's go and get like an old school image i don't know what's what's trending right now if i go to reddit [Music] i don't know let's just let's just do more formula one ones um so what's called irene racetrack what about this image so this is 997 by 561 so if we go and save that one down you could use any image you want right i'm just messing around and plugging in random images but if we go and do this i've just gone and dumped in an image of the race right and again we can go and take a look at results that's gone and been up sampled so again ridiculous results and like i mean you saw how easy it was so i'll include these instructions inside of a separate github repository with full credit to this github repo because he's gone and done all of the work or they've gone and done all of the work but really all we had to do was those four steps so you clone the repo using git clone download the model from google drive or buy do drive install those dependencies so pi torch crude and then open cv python and glob2 and then it's literally that simple so you grab the low-res image throw it into the lr folder and then run pythontest.pi you're done and dusted pretty cool right relatively straightforward to get up and running but on that note that about does wrap it up let me know what you thought thanks again for tuning in peace thanks so much for tuning in guys hopefully you enjoyed this video if you did be sure to give it a big thumbs up hit subscribe and click that bell and let me know what you thought of esr again are there any other games we should be taking a look at what did you think of the tutorial in general did you enjoy it did you get it working thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 94,400

Keywords: super resolution gan, super resolution, super resolution deep learning

Id: 67hqsP7fhvQ

Length: 21min 24sec (1284 seconds)

Published: Thu Mar 10 2022