How to REMOVE TRACKING MARKERS with the MATRIX node in NUKE | Comp Lair: Live Tech Corner S01

okay so today we're going to talk about something that you came across all of you came across at some point in your careers or this is exactly what you do every day but this is something that can follow us independently on many years you have in your career so what we're gonna talk about today is alternative ways to get rid of tracking markers like this and because these tracking markers sometimes have like very different shades in color you can sometimes extract them easily by flipping the channels or by pulling a key or something like that just to get the mat out of it maybe you can maybe this is not the ultimate example of the things that i'm trying to talk about here today but you know what i'm talking about when you have tracking markers that have like this weird mixing color that it's not easy to extract them with a normal gear so we're going to talk about alternative ways of cleaning plates with tracking markers like this and one of these ways will involve matrix multiplication via a color matrix okay so we're gonna do a quick recap on matrix multiplication i'm not gonna get into what a vector is what a matrix is it's more about just focus on the multiplication part multiplication between two matrices or a matrix and a vector which is how color will be interpreted in this context but this will be one of the things that the technique that i'm gonna show you will use and that's why i want to do this recap with you so you have matrix 1 matrix 2 in this example and the result will be this and how we get this result well first of all in order for two matrices qualified to be multiplied by each other either two matrices or a matrix and a vector which can be written also in form of a matrix the number of columns of these first matrix have to match the number of rows of the second one if they don't match effectively in mathematical notation you cannot multiply them if you put this on the software maybe you'll figure out that what you mean or what you meant is the other way around but in mathematical notation you cannot do that so in this case we have one two three columns and we're going to multiply by the second metric that has one two three rows so they qualify for that so we can advance with our calculations so the way that the multiplication will happen is we're gonna go on the horizontal direction on the matrix one multiplied by the vertical direction on matrix 2 and that result will be placed horizontally on this one here so what you're going to do is you're going to multiply 1 times 0 plus 2 times 2 already on the second term of the first column of matrix 2 plus 3 times 2 and this result will be the first term on your resulting matrix so 6 plus 4 is going to be 10 right so we've exhausted the first column in the same here we've exhausted this first column so let's jump now to the second term that is part of the second column still in the same row so we're gonna maintain this row as well let's do it so one times one is one two times zero is zero and three times one is three all of these combined three times one is going to be four so there you go you have four then you're gonna jump to the other column but still maintaining this row okay so again one times two it's two two times one is two and three times 2 is 6. all of these combined is going to be 10. and then because you already exhausted the columns in here it's time to change the row and we're going to do the same column by columns so what we're going to have is now 2 times zero which is zero plus one times two which is two plus two times two which is four all of these combined is gonna be six and you're gonna do the same as before until you exhausted all the other columns and rows and this will be your resulting matrix another thing important to recap i would say is the dimension of each matrix after we already know if they qualify or not for multiplication but once we know the dimension of both matrices that you're multiplying it's possible to know exactly what would be the dimension of the result of this multiplication so in this example we have three rows times three columns right and now this one is the same so to qualify what i said is if the number of columns is the same as the number of rows on the second one they qualify for the multiplication okay otherwise they wouldn't be qualifying and the remaining numbers here once you put them together they will be the result dimension of this multiplication so the resulting is going to be another three by three matrix so this is very handy for example if you have matrix one if now you multiply this by matrix two okay so if you had this matrices this one will be possible why because this is a matrix two by two and this is a massive three by three so the number of columns here is not equivalent to the number of rows in here so this cannot be multiplied at all let me give you another example so if you have matrix one then you wanted to multiply by matrix two okay so this will be three by three matrix and this will be three by one so now this number is the same as this one so they qualify for this multiplication and the resulting matrix will be one that has dimensions three by one and once a three by one is the same dimension as this guy so this is the foundation on how you multiply a matrix by a vector and this is exactly what happens with the caller matrix so i'm going to give you a real example okay let's say that you have a matrix and then you have a vector that for example it's a color i'm going to call matrix two let's say the color yellow which is one in red one in green and zero in blue so if this is yellow what will be the result of this yellow once you multiply this let's do the multiplication let's do this like this multiply it by this like this so let's do this by hand so you get the hang of this so two times one plus one times one plus one times zero see two times one one times one and one times zero so we've exhausted the columns on this one otherwise we will have another column here so it's time to change the row so it's going to be 1 times 1 plus 3 times 1 plus -1 times 0. we don't have more columns change the row then okay so now it's going to be 2 times 1 plus 1 times 1 plus 3 times 0. and this will be your resulting matrix that we already established that is going to be a 3 by 1 matrix so two times one this is gonna be two this is gonna be one and this is gonna be zero okay so two plus one plus zero this will be the equivalent of three this will be one this will be three and this will be zero so this one is going to be four this one is gonna be two this is one and this is zero it's gonna be three this will be the resulting color in red green and blue okay because we can write a color as a vector like this okay let's go back to nuke and let's put this matrix on this color to see if that's the result that we have okay so let's put these numbers in so our constant will be our yellow color which will have a vector defined like this which we know that is yellow color because this will be red green and blue so 1 1 on the first two terms will give us yellow cool so now let's put the color matrix and by the way this color matrix has nothing to do with the normal matrix node here this one they operate completely different and they're not the same this matrix here we're not going to cover today but this is the convolution matrix which operates in a completely different way and achieves completely different things so it's not the same so color matrix is actually a normal matrix to do normal operations such as multiplication which is the one that we're seeing at the moment so let's put now the numbers that we've defined randomly so two one one one three so our result was three four three let's see if that's what we have three four three so now you understand how the color matrix works a lot of people don't know from my experience how this works now you do so it's a good thing so let's now use this color matrix to do what i've laid out in the beginning and that's the actual core of what we're discussing here today so i have these plates i have two versions of this plate one denoising one with a normal noise in it and um you want to clean things always under the nice plates for obvious reasons so you're gonna put the noise back on top if that's what's needed or you're going to deliver this if that's your role you're going to deliver this to comp with no noise at all and then they'll put the noise back on top so let's now go through the channels and let's see what's the cleanest one by cleanest i mean where do we see last tracking markers and i can see that on the red channel which is the one that i'm at at the moment it's the one that we see last so let's see if we can make it even less visible how with the color matrix and one thing that i didn't say before is if you put the diagonal here with ones this will be called the identity matrix and this is the starting point for you not to have any type of transformations as soon as you put this one plugged in your image there's reason for that and it's very useful i'm not going to cover this at the moment because it goes outside the scope of what we're trying to achieve here so now that we know how the metrics work i will leave the calculations just for a second because i've been playing with color matches for so long now i develop like a field when i mess with these values here so the goal here is to put these tracking markers on my red channel which is the channel that i'm at even less visible you will also develop this fill after you play with this for for a while but if you go up and down with the numbers you'll get more separation or less separation in this case i want less separation on the red channel so let's see what this will give me and by the way i'm just going to apply this on the red channel that's the channel that i want to change so this matrix in here each element will be red green and blue red green and blue red red and blue that will be multiplied in the way that we just looked at before so if you go up and down with the numbers you will see that this will change of course the channel you're in so the goal again is to have even less separation between these tracking markers there's a limit of course to what you can do we're gonna try to push that limit now that i have this what i'm gonna say is this is the cleanest one right so what i'm gonna say is all right i'm gonna put the other channels as clean as this one too so i'm gonna use a shuffle and i'm gonna say now that my green channel that before it's this i'm gonna say that now this green channel will be the same as the red channel so we're gonna say that the green is gonna be the same as the red so we swap those channels so now if you look at here both red and green they are the same you see they're changing here but in reality they're not changing here at all one thing that i have to do is to match the luminance of the green channel as it was before so i need to go on the green channel here which was my main image and i'm gonna say that this luminous needs to match now with this new green channel of course you can have ways to do this a bit more procedurally but we're gonna do this manually for now so you get the idea of all these things work so i'm gonna put a wipe and i'll try to match them as closely as possible okay so now it's matching now i just want to apply this on the green channel so now i have my red channel here green cleaned as well and now i have to do the same for the blue because the blue will still have this marker so i'm going to use another shuffle and i said that my blue is going to be the same as my red again red is my base and i'm going to match luminous of this new blue channel by what it was before so i'm going to put another grade apply it just on the blue channel and i'm going to compare these two so they're very close but this needs to go down a bit okay something like that cool so now i have my red green and blue both of them as clean as possible so here we have it we don't see the markers anymore almost this is not the end of the story though so now what we want to do is we want to apply this result not on top of everything but just where the markers are so how can we isolate the markers well we have several ways but what i want to cover here today is the color matrix node so i'm going to open another color matrix and i'm going to try to have as much separation as possible overall so i will have an easy way to try to extract and again i developed already a certain feeling for these things so i'll try to get as much separation as possible you have to go really easy on these values by the way you see that i'm on the decimals here by playing with the values i discovered that is actually the best result so if i go on the blue channel i have total separation cool that's exactly what i want okay and by looking at the channels i can see that i have some fainted values in here so i'm just gonna open a grade i'm gonna try to crank those values up a bit okay so this is my math now i want to basically apply this result that i've cleaned just on those regions so i'm gonna use a key mix i forgot to just say that this path is only valid on the blue channel okay so i'm just gonna shuffle the boot channel like that and let's see all right it seems like this mat is a little bit too short so i'm gonna grow it with an erode maybe the caution filters we still have this ring with colors that they're not so pleasant so because the red channel is where everything is being referred to let's see what else we can do to clean this even further what if we put a blur so these things will be even less noticeable i want to apply the blur just on the red channel but let's see what it does so if i blur it blur it will be a point in which we don't distinguish this as much let's see if this is enough yeah there you go it's enough so before and after before and after so now if you look at the channels they're much cleaner still not clean 100 but they're much much cleaner so now you can apply a technique to basically even out the screen i have my own tool that i'm not going to get into how it works at the moment but this is based on the ibk stack basically it's called keybody and what i can say is from this plate i want to basically create an ibk stack like i want to erode that region so i can get all the hand inside that mat and then i'm gonna patch it and now i can create an even screen so i say calculate there you go so now it's completely even and now if you flip between the channels the trackers are completely gone and this is a semi-automatic way that with some luck if the lining in the plates are similar you can get this setup to be passed on your teams and different people to get rid of it semi-automatically if you want to go and add the noise is to basically minus the denoise by the plate and then you'll be able to add this original noise back on top as a plus okay so now we have the noise back back on top so now you can compare this with what it was before and it's completely even you're not losing any detail in here you'll have all the edges and all that stuff by the way this tool also allows you to choose what's the color that you want to put in the even screen i've put this as an automatic way as you've seen but can actually choose the color any color i want any shade that i want any value that i want it's possible even different colors i want i can do that but this is optional the cool thing here is that you have all the channels cleaned completely from the tracking markers and the color matrix allows us to separate the colors so you can use them as mats so the explanation in the beginning was a little bit overkill but i think it's useful for you to start understanding how these things work to achieve different things and as you can see this can be used for simple things like this with a very good degree of success and now that you know how color matrices work you'll be able to out of a certain shade what would be the matrix that you would have to apply to the image in order for these trackers become blue this might be a tech challenge actually one of these days

Channel: Comp Lair

Views: 655

Keywords: linear algebra, linear algebra for cg, tracking markers green screen, tracking markers vfx, remove tracking markers nuke, comp lair, the next level, nuke tips and tricks, vfx geek, vfx, advanced compositing, advanced nuke, nuke tutorial, nuke advanced tutorial, compositing, nuke, maths, foundry, Corridor Crew, Movie VFX, hugo's desk, cgmeetup, vfx guru, marvel, ilmvfx, weta digital, framestore, mpc episodic, lumapictures, trixter film, imageworksvfx, Rodeo FX, Digital Domain

Id: ebn9Lw84hLc

Length: 16min 1sec (961 seconds)

Published: Thu Nov 18 2021