Independent Component Analysis 1

all right so we're gonna continue talking about essentially the SVD and as I said it's kind of magical in many ways because of all the things you can do with it and we're gonna shift gears a little bit to talk about what's called independent component analysis okay so let me write that up here you gotta be careful where you stop that way everybody awake yet okay all right I see a for short so you see you know we have SVD we have PCA we have P OD now we're just gonna add ICA and here's my claim you already know this one you know this one well because you do it all the time let me give you the classic example I call the cocktail party problem alright so here is the basic concept we'll illustrate it through a little cartoon it's like cocktail party starts fancy kegger we're at a kegger and there's two groups of people talking alright and here's the idea here's the experiment steps you have a conversation one conversation two people conversing suppose they place two microphones in the room okay and so here's the deal two conversations happening I can record them here in here so at microphone one in microphone two and what I have here is two signals being produced two acoustic signals let's call them s 1 and s 2 in time however when I record them right I get a mixing of these two signals right here at these microphones I have a little bit of s1 and a little bit of s2 and my flashing back there okay stopped alright okay does that make sense here's the mathematical problem you want to do can i with these two measurements actually separate back out these two signals okay that's the mathematical idea now we do this all the time right we can be in a conversation you hear a word and you can almost instantly train your ear do we do this right you can kind of almost eavesdrop of what's being said over here even though you have all this signal over here you can actually extract that signal out provided it's not too loud and then everybody's speaking sufficiently high volume right so this is the idea of what we want to do now from a mathematical point of view then you say I have two signals and let's take recordings at microphone one let's call the X 1 and X 2 and I take these recordings and here's the idea my recordings at the microphone 1 & 2 come down to be something like this I'll have signal s1 I have signal s2 and I record some linear combination of these two signals at microphone 1 and at microphone 2 so what I have is these guys and what I want is s1 and s2 the parameters a IJ are the mixing coefficients okay and what I'd like to determine is how to handle that a couple things to note right away is part of the deal here is you don't know what these A's are right so think let's just do a little count how many things do I not know I mean not me personally because that's - it's like infinity but here in this problem you don't know this you don't know that you don't know this you don't know that you don't know this you don't know six things six things six unknowns and two constraints right so this is kind of bad deal for you because typically we think about oh well you can't solve a problem like that because you have too many unknowns and not enough information to actually solve this okay so we're gonna talk about techniques to actually figure out how to not get these but to approximate these okay alright because if you knew the A's then getting the original signals is trivial right this is really just a something like this and if you want your signals out you just you have something like this and if you knew them at a matrix that's easy but you don't know the may a matrix and how you would you determine the a matrix by the way so the a matrix depends a lot upon the actual placement of the microphones as I move them around the room depends upon what the room shape looks like and we're ignoring a lot of effects like for instance you know acoustic waves bouncing off the wall and so forth okay but you can see if you could find an algorithm to done this you could place microphones in the room and tune in to different conversations that are going on now by the way uh where does this apply beyond just sort of cocktail party conversations well we've already talked about many examples but here's some that you would see that are easy ways to think about conceiving where this would be important radar detection if you look at all the radar detection stuff we did we are saying we send out a signal bounces off a plane and it comes back well what happens if we send our signal out and there are multiple aircraft out there okay so stuffs bouncing off more than one target let's say and part of the deal is we would like to actually figure out how to pick out different targets you know those we want our radar to actually pick out all the targets in the air and one of the key things that's going to happen here when we try to actually apply this method and determine those A's is that we're gonna assume something about these signals in particular we can assume that the signals produced from different targets like the airplanes or the signals produced from different conversations are gonna be statistically independent okay if they are statistically dependent we're kind of in trouble but if as long as we make that assumption that look this conversation and the pitches and the voices and what they're talking about somehow is completely different than here then we have this idea of statistical independence and that's going to be the main thing it's gonna allow us to basically get out these mix mixing coefficients another application where it's independent component analysis is really big not egg's EEG so EEG is basically put a bunch of I don't know electrodes things like that on the scalp which measures brain activity essentially so part of what you'd like to do is if I have enough of these on the brain I some kind of overall activity but really what I'd like to do is separate out the different signals that are actually going around in my brain okay so different parts of the brain sending different streams of information if you assume that these different parts of the brain are actually assuming statistically independent thoughts I don't know whatever things why not thoughts all the time but things going around in my brain how do I pick out what is there okay and the final one which I love so this is really much more practical in Terminator the new one so you saw that in Terminator they found this hidden signal upon all the signals that were sending out to control all Skynet and they said there's a there's a there was a hidden signal on top of it so somehow someone in this class in the future finds this right it could be mad maybe Matt's in the future and he found this signal and he did it basically used the method SPD and he found this signal that's how they found out how to shut off some terminators anyway you see how important it is I know Maria yeah maybe you're not understanding how important that is because you could turn off a terminator with with your skills that you learned today in class everybody pay attention you're needed in the future all right any questions on you see what happened here we're just taking signals or trying to separate them all right and we need a mathematical technique to do it all right so that's the concept so let's try putting this thing and some kind of framework mathematically that allows us to compute this and then we're gonna focus on a very specific example of how to do this which is gonna be an image analysis problem and I'll present that a little bit later okay here's more a more formal framework for this okay all right here it is given n distinct linear okay and distinct linear combinations of and signals okay determine the original and images okay all right I think so so that's the statement I have the idea is that somehow I have n signals all being mixed together I have in measurements of these things so somehow they are in linear combinations right so wherever I place measurements devices things like this in my system I read off in things and then with those n measurements can i in fact infer the original and images that are actually mixing J does that make sense so in a more general setting than just this two-by-two system what I really have is I have I have some X of J which is going to be my measurement is going to be some kind of linear combination like this so I have some Jeff measurement and it's a linear combination of my signal s 1 s 2 s 3 S 4 all the way to s of n okay or Express the matrix form where I calculate all of these matrices what I have then is this the only thing I know about this equation is X that's what I measure and these guys over here we don't know and I want to find it so mathematically self-consistent way to approximate what these are okay it's gonna be the best and by the way I want to do this in such a way that I make the minimum number of assumptions that I that I can okay so in other words you'd like to make it as general as possible how do i reconstruct the original signals based upon my measurements of the mixed signals with as few assumptions as possible on how they mix okay because if you make it too restrictive most likely is you have an experiment that you can't actually do right so you have to try to make it as general as possible to make this work all right this is called an opponent component analysis that is what you want to do you want to pull out the independent components that are sitting inside of this thing from your measurements so we talked about sometimes images sometimes signals but you can actually see that this is has a much broader impact which is you give me data sets of measurements what's to say that there actually isn't all kinds of processes that are actually coming into your measurement that leave a big signature but may be completely statistically independent of what you actually are after right so you get some measurements and it's polluted with some other measurements it's not necessarily a bad thing but you have to figure out how to withdraw those things out of this thing okay that's the that's the that's the purpose of this so for instance if you're an atmospheric science maybe you're getting some measurements and you're interested in one kind of behavior out of the system can i quantify certain thing but when you do the measurements you're actually getting a bunch of other effects they may be statistically independent and you'd like to just remove them so you just isolate what you want all right do this in a graphical way let's think about it this way one of the things we had talked about with principal components so we want to talk about PCA and how does it relate to ICA besides just the first letter change okay they both have this idea of component analysis one of them is principal one of them is independent what's the relationship so let me draw a series of figures for you to graphically illustrate how we want to think about this suppose I take some measurements this in fact could be something like from camera one that we did on the homework and I just plot position data that I get and maybe it looks something like this all right you guys like that sound okay all right a little bit annoying that's all right but suppose I got something like this all right took some measurements and what do we do well you know what we do we take this thing we subtract off the mean so we put ourselves right in the middle here somewhere and then we do this SVD and we say tell me which directions are essentially the most important in this system and when we do that what you would find for something like this is there is a direction probably it's the whatever you know we've talked about this as being what's gonna pull out for you is the semi-major axis of an ellipses it's there if it's two degrees of freedom the SVD pulls out major axes okay it pulled out the biggest one first second biggest one second third and so forth and it pulls them out in orthogonal directions so for instance if you were to did an SVD on that data set what you are gonna do for your homework you'd get something like that and that's there's my principle component and has some value Sigma one that kind of gives the strength of that component and then for this data you can even go to Sigma 2 which would be orthogonal to it so this is where we need red pins but fine can you all see that there we go there's your Sigma 2 does this principal component decomposition tells you that most of my energies in Sigma 1 I get a little bit in Sigma 2 and this is a simple two dimensional problem but you can imagine you can have higher dimensions and it would just project everything down to some appropriate basis okay good that's what we do that's SVD and pca and p OD in a nutshell that picture yeah no no do you like that picture she was giving like so I was like ok all right well anyway maybe you don't like my drawing so it's okay I'll try to improve ah let me give you a second data set and let's talk about doing the same kind of idea but I take some measurements here's what I get ok I'm gonna make noise okay suppose there's my measurements so some other experiment and this is what I get I project all this stuff down here's what I plot okay now first of all part of what we're training our computer to do do everything automatically for us so sometimes you might say I got this Fink feeder down I got to do is take the day to put it in say SVD and just look at Sigma 1 Sigma 2 and but part of there's a value in saying okay now this is make sense and we project my data let me put my my my principal components on this and so forth because if you were to just blindly apply your SVD right now to this thing here just put the matrix in SVD pop out your orthogonal modes puff out your Sigma 1 direction which is your principal axis it probably looks something like that and you could ask yourself is that a good representation of the data and I hope you would say the answer is no actually there's nothing along that principal component at all you've somehow got data lined up here you got data lined up here you got nothing in this direction and so what's somehow telling you is that you've missed something kind of important about this this is a is miss analyzation a word I don't know if it is but you've that's what you've done it is now you've Mis analyzed that if you've that's what you pull out that's a straightforward pca application of that data set all right so what's the problem the problem seems to be that in fact there's really sort of two principal directions there is a data set lined up here in which you could extract out a principal mode here and a principal mode there sigma1 sigma2 there's also a data set along here which you could extract out you know let's call it Sigma 1 bar 2 bar this is enough sila stration of the independent component analysis what you've got here is two data sets that seem to be independent of each other clearly stuffs happening here stuffs happening here two different phenomena and in some sense what you want to do is do an SVG of this data and SVD of this data you want to separate them okay this is what it might look like if you looked for instance at the data that would be generated at microphone one in microphone to the conversations happening here and the conversations happening here maybe you have you know the one dude really loud guy it's always at a party he's usually goes outside and sets on the porch and everybody in the neighborhood can hear him anyway I live I live close to here and I know this one guy there's always one of them at the party and he's always hot so he always goes outside and he wakes everybody he's always got the beer and his voice is super super loud so it's hard to see if anybody would else come in but you know you there's also the girl was the higher pitched voice it also is outside and you can hear her really well like I said I live close to campus so I know these things so let's make a go here alright so for instance if you were to record these things and they're dominating the conversation and it's about 1:00 in the morning and they're just super loud out of control because this thing controls their volume right here and after about 10 of them their volumes are just way up okay so they're talking really loud you're picking them up here but you know ultimately these are two very different voices two different Cadence's two different measurements that you take here and when you look at the data from x1 x2 and start projecting out this data it's a linear combination but you might get something that looks like this here's the one person talking here's the other person talking to decompose them is to mix their sort of voices in some sense so what you want to do is separate out the signals and once you separate out the signals then this and this makes sense okay everybody understand that graphic it's really important to understand that that's the key thing right there so whenever you have data and if you can in fact project it to some low dimensional thing like this you first want to take a look at that data does it make sense for you to just simply do an SVG straight up on the data because in fact if you did like I said you'd be getting a vector like that which is Sigma one which it actually doesn't really tell you what's what's going on in that data set they would be like somehow representing the S as a function of time in volume or pitch or something like that you'd make up some variables that you're trying to pick out of that signal yes oh well okay so actually no you don't know the S is you just have some measurements and the idea here is I've only given you one variable but you might be measuring from microphone one let's say volume versus frequency for instance you could take two quantities that's why it so I made it two-dimensional let's say to represent so for microphone one let suppose you measure two things from conversation one two things from conversation too so you don't I mean you don't have to measure one if you do to this the death the graphical dissipation this really doesn't make that much senses are there but I'm stretching it it's just a cartoon so you shouldn't take too seriously just like my stick people over there you know the more recordings you have the more likely you are so there's a point in which you over determine the system right and so now you're checking for consistency but if you were to keep taking measurements there you could then for instance in that simple example if you just did this you have here to constraint six unknowns you can imagine I put four more mics on that's four more constraints then I could potentially pin down the A's well actually I'm adding more amore values right to so potentially you could however start refining it more and more as you get further down what you'd really want to do instead is get more information in each microphone right so the a 1 and a 2 a 1 and 1 and a 1 to stay the same but you just keep adding more data to nail down those parameters okay let me finish off with one last cartoon here normally that's a good okay.i that's kind of normal normal distributions are bad I was gonna say normally you don't were the last thing you need right now Gaussian variables Gaussian distributed variables kill ICA here's why suppose I have a Gaussian distributed variable ready more noise so tell me suppose I have really two independent signals in there and I get them to be Gaussian distributed which way is Sigma 1 which way is Sigma 2 right you've got this inability to extract out principal components at this point ok is that it you rotate that any number of degrees I want and it's still kind of look the same so what ends up happening if you have Gaussian distributed random variables pca breaks down I mean sorry ICA breaks down so one of the assumptions want to make here is that we're not going to have the ah seen distributed variables okay and by the way we're gonna use this extensively to do our image analysis okay questions on anything so far everybody good so far you guys want to come over hang out at my house no okay not yet all right I've made a convincing argument okay now we're gonna do a very specific example the specific example what's gonna be awesome I would assign it with homework but you have these you know these homeworks now I got a report to do so but we're gonna work this out completely in class and it's going to be I think a really nice problem here it is by the way we're all familiar with this as well and what's amazing is how well we actually handle this here's me there is a beautiful piece of art I don't some people maybe a mountain oh that's art okay yeah and I'm looking at this beautiful piece of now alright now it turns out this piece of art was done by someone very famous so what they tend to do then is put it behind glass okay that's glass can you see through it right right oh you know what you do in your cartoon you listen it like supposed to show you that it's glass okay glass in front of this thing interesting ly enough first time I ever saw the Rosetta Stone in the British Museum you could I was very hard I wanted I wanted to give elbows to these people who are touching there was out of something no it is they're just getting a grubby hands all over it I was like I couldn't believe it just because it's a rock they think they can touch this thing anyway they now have it behind glass I'm so happy but here's the thing I'm looking at this image and there's a window back over here somewhere so it's some light source over here so the reflection so when I look at this image the light not only lights up the piece of art but the glass itself reflects a good portion of actually the light that come so in other words before I even get to the art back there it's reflected off the glass all right so just take a look in fact out the window on any day if you can often see you see outside and you see your reflection sometimes where you see the reflection of your room right there on the glass okay so that's the problem we have when we take this picture if I took a picture here's my iPhone and I take a picture of this I don't just get that image I had the image plus maybe there's some kid over here touching the Rosetta Stone I want to kill and he's in the picture now okay and I have to edit that out because it just makes me angry every time and it's not good for my health okay so this is very you see how important this is promise all right good I just want to set the bit you know terminators we're working on solving terminator problems Plus this look at all the skills you pick up in this class at the end we're gonna we're gonna make a big list of all the things you can do like destroy terminators remove kids touching overhead of stones it's gonna be awesome that's exactly it you could work for CSI after this - I'm gonna add that to the list because the murderer is right over here all right or the art theft let's go classy okay all right so there we go I have an image I have the reflected image and basically I want to extract one from the other now interestingly enough we are quite adept to doing this on our own you can look through the window you don't even see the reflection that's clearly there like we just trained ourselves to look right through it but if we want we can actually all focus in on the reflection we can pick out images amazingly well okay try that sometime take a reflection what focus just on the reflection you cannot actually make out them almost like it's a mirror not quite as good but you can you have an amazing ability to extract out that image that's the reflection then you can also just say I don't want to see that I wanna just look through to the picture behind it and it's amazing how you almost don't see the reflection that's what we want to do ultimately there's a kind of a principle here which is some people are trying to teach machines how to do virtual scene right and how do you extract these images because we can do it without any problem it seems and that's that's going to be the concept so let me turn on the overhead camera here but let's see is that overhead cam oh I got to pull this down right so that you guys confront can see this well we're gonna get fancy here alright are you ready are you you're not ready for fancy they that it's past night now alright alright I just turned to display on yeah alright so they have to go very far I went right to my office to get you two examples of pictures okay so let's do this okay and do I need to do anything else project local is that right all right Oh document camera there we go there we go okay let's see if it switches over perfect okay do you want to switch over the turn off the lights in front here yeah Oh - okay ah what do you think what was better this do the other one that let's do lights on real quick all right yeah the same it's like the dot it's like going to the I pointed this or this and you sit there to answer that question for ten minutes and they say okay here you go done oh you did better fine all right all right here is a picture an image if you've paid attention while you've some of you've been to my office many of you have not but there's a picture up on my wall have you do you know this one have you seen it did you even look at my wall am I just wasting my time with art on you guys you guys have seen it cuz you've been in there so many times Edwyn really come on you don't have picture I have this picture of my wall here is in fact a great classic mythological theme the judgment of Paris here's Paris right there and he's got the Apple the Golden Apple for the fairest oh well nothing gets a goddess gone then - no she's the fairest so what we have here is a contest this was dropped off at a wedding party of Peleus and Thetis because discord didn't get invited she was a little ticked off so she throws his gold map on the table says just for the ferris and of course these three lovely ladies over here all believe they are the fairest we have Hera and we have Athena and we have of course Venus and then they have a contest aside oh oh nobody wants to decide this including Zeus he's like wants to stay out of this because this is really problematic it's not like let's have some dude to side this there's a guy Paris it's over there and here's Hermes and they say okay you are gonna decide who the fairest is right so there's there's the Golden Apple to give and so they each promise him something Juno promises amor Hera promises him he will be the greatest ruler of the land rule over vast empires Athena promises him to be the greatest warrior of all time sort of kind of a promise that was made to Achilles really but you know he could have taken it instead so and then of course then Venus comes along and promises the most beautiful woman in the world oh okay well here so so there you go that is the the scenario and of course he picks up Venus that's the this is the the tragedy of the Trojan War because the woman that comes out of this deal is Helen of Troy so he goes and steals her brings her over to Troy and then of course a big war so this is by this guy John Flaxman so he does his painting so now what I did is I took a picture of this now what you can see here that's my office window there's a reflection of it doesn't show up quite as well here but you can actually look in the reflected image if you look over that you can actually kind of see the outlines of my books up here it doesn't show up so well here but if you look a the notes online you can see a little bit better and now here's the window this is looking out towards where the fountain is that's a reflected image so what I've done is two experiments so I'm all about experiments and let's see if they work I took a camera I took a picture of this in my office and when I did it I put a linear polarizer in front of it so a linear polarizer took the picture now what we know is that the way that light bounces off this glass depends a lot depoe it comes off in a polarization dependent way okay so if I take that linear polarizer and I rotate it by 90 degrees I get a slightly different image which is here it's the same thing however it looks like the same thing but you'd actually get slightly different polarization states and if you actually look carefully at the images they're slightly different so here's going to be the deal I'm going to try to separate the image in the back from the image in the front which is the reflection I have two pictures here the reflected picture which is represented by this window plus you can't see it here but there's my books up on the wall that's image number one from the reflection image number two is what's underneath here which is the Flaxman illustration of the judgment Paris okay so of two images superimposed what I've done is given myself one more data set by doing the linear polarizer and my objective is then just take these two images on separate okay kind of cool right so you see how this might play out right I mean this is something that for instance you could almost you can almost build it indirectly into just you pull up iPhoto on your Mac or any of these software things you say separate image and because we're going to talk about is almost an algebra algorithmic way of just pulling these things apart okay all right so that's going to be our objective for this week so today I'll talk more about the set up the philosophy of what we're going to do Wednesday work out all the math that we got to do Friday apply it and clean up the nice plan of attack I think so and you can imagine that you might do this actually not with pictures but with real data give some data and I want to just take the data and just separate out the data so in that case this is this is just philosophically keep in mind this is not really an image processing thing this is about data separation the images just happens to be data okay we're selling it this way and it's really easy at least from this data separation point of view to see it as an image separation because then you can see oh how well did I do oh really pretty well or not okay and of course just like the experiment in class last week III think it might work Matt will probably say no there's no way he's gonna work we'll stay on Friday we'll see on Friday that's that's all I can say for now I have no idea because I haven't actually done it I mean if I'm kind of have the images I'm but I went on my algorithm to it yet okay so let's talk about what we're gonna do then with this so is everybody okay with the premise of this thing okay so let's try to set up this problem and learn how to think about this problem it is no different than we've been writing down here two images need to be separated when I read this at my camera I have a linear combination of these two images by taking that linear polarizer shifting in 90 degrees what I've got now is two independent measurements I could have gone to another place but that's actually for the for the image thing you want to stay in the same place take the picture rotate the polarization take a picture extract out I want to take away want to have that nice beautiful Renoir in my frame okay actually does run off a picture people he does right a little bit but they're always we got hats and dresses and stuff my people are naked you could tell okay see all right anyway Adam and Eve in the Garden of Eden that's what I call that work all right nobody's appreciating my art you know what we're gonna do field trip to the Sam so if you guys can't if you guys can't appreciate good art we'll go down there British exactly wanna see whoever can touch the Rosetta Stone gets an a Josh you're strong start working protect glasses thick you gotta break through it and didn't touch that stone but don't worry it's all worth it a few days in London jail 400 pays off I can't pay airfare okay so let's talk about separating this out and let's talk about the SVD and how we want to apply the SVD to thinking about this problem by the way let me let me make one other comment there are other ways to do independent component analysis in fact there are many ways to do this there are a lot of statistical methods that have been developed by Casta tist achill like maximum likelihood ratios and estimates these kind of things all play roles in doing independent component analysis so I'm not saying that this is the only way but what I am gonna say is we're going to take this and given what we've been doing with the SVD in the linear algebra the sort of the more of applied math way of doing it let's use that as the tool of choice in this thing so it's me SPD and how it's going to impact our ICA okay everybody ready I have two images I have two measurements so s was a vector contains all my image information X these are my measurements that I have x1 is with one polarization state x2 is with the other state okay that's the setup and here's what I want to assume and this is gonna be very important for us first s1 and s2 are not Gaussian distributed images so it turns out this is actually a good assumption most natural images are not Gaussian distributed okay we're gonna make use of this fact in fact what we're going to make use of is higher order moments in statistical distribution to get us what we want that's typically what you do with independent component analysis it's all about the higher moments you use the higher moments to discriminate directions okay - they are independent what does it mean for two variables to be statistically independent well if you remember from our definitions in class from lecture I think number two of this class the distribution the joint distribution of the variables s1 s2 is just product it's separable of the two okay there's going to be the assumptions we make all right now what I'm gonna highlight here is the cartoon of how this works that's all I'm gonna do class Wednesday we're gonna formalize this cartoon and put mathematics on it okay but first thing you have to understand is the cartoon if you don't understand the cartoon the math isn't gonna make any sense to you okay so understand this cartoon what does the cartoon gonna be well the cartoon to understand this is the fact that we know we can take any matrix a and do an SP decomposition on it decomposition on this thing so what's that mean well means take a and basically do that to it we've talked about is V and U or unitary so they're just gonna make rotations Sigma is a stretching matrix you stretch according to the principal components okay so that's what this thing does so think about it in the context of this X is equal to a s so X is equal to okay so let's talk about what this does to measurements and let's start off with a simple assumption let's assume that s1 and s2 are somehow uniformly distributed variables so let's start off with our original images sorry s at this point they're unmixed what I'm assuming is image one is given by some uniform distribution this way he understood in uniform distribution a which they are basically not mixed or statistically independent so I start off with a distribution okay so nose I can live anywhere along in this plane this is where my image plane is what does the SVD do first thing it does rotates it stretches rotates what does this look like take this thing rotate it some amount so one go from here to here that's under the action of V Star stretch it that's the action of C rotate it again okay there's supposed to be the same size but I'm running out of board space so you all good with that and then so so to get from here to here you did the Sigma to get from here to here you did you this is what you actually measure right this is the ideal if you actually have the images this is what this matrix a would do to your image so not what you have what you have is your starting up here you're making this measurement right there okay so here is the idea of independent component analysis I want to take this mixed image that I have in this framework I've got to figure out how to separate them the separation problem then comes down to simply this how do i back this process out in other words how do I take this rotate it down to that how do I take this stretch it back out to uniform distribution in each direction and then how do I rotate it back to here and this is gonna be my approximate image because when you think about doing this you kind of go like well that's kind of I'm undoing you so it's you star remember you star it's a it's a unitary matrix so you star is its inverse and then I want to that's what I want to do I don't know the matrix a though though by the way right I don't know the matrix a that's the whole point so normally I'd say Oh take the matrix a SVD you do not know what the matrix a so philosophically what you've got to think about in this process say I have a distribution like this how do I find that well first you got to take your data the first thing you got to do is figure out that's kind of like your Sigma 1 and Sigma 2 directions so what you really want to do is when you look at this this is the maximum variance along that direction next direction has a smaller variance so the first thing you gotta do is pick out where that direction is we'll take your original data find out where the maximum variance is make that some angle okay that's gonna give you some angle here let's say and then make this orthogonal to it so just make this so you determine it directly from your data - oh well I can kind of extract what that angle is of this stretched parallelogram that I've got okay for uniformly distributed data so I know how to rotate it back to here so you just rotate it back to here now once you get the strengths of your variances you can actually how did we compress this you remember you compress this by having sigma1 sigma2 along the diagonals so once you know your principal directions strengthen your principal components you know how to stretch it well you can actually predict what the strengths of Sigma 1 and sin or two from here are and then you stretch it back once you stretch you back these ok it's still not done because they're still mixed in this representation the last thing you need to do is now that you know this angle you rotate it back to here and once you rotate it back to here and by the way when you do these steps you're gonna have to use higher moments to figure out how to rotate it back and we'll talk about that on Wednesday but then you rotate it back to here and what you've essentially done is you've backed out the vd you've backed out the you the V and the Sigma now once you have those then the process is easy you approximate each one of these pieces of DSV D and then all you got to do now is say oh I want my original images that is take my data apply these matrices that I've built that are approximations to what should happen get out the images and that is the separation process so we're going to talk about how to construct these UV DS ru Sigma and V's on Wednesday on Friday if all goes well I'm gonna take that raw data and either fail or come out of winner one other - there's no there's no middle ground really well anyway have a nice Monday nice beautiful day I think it's beautiful for my name Tuesday and then Wednesday it's gonna start getting bad and rainy it's not I warn you right now

Channel: Nathan Kutz

Views: 17,995

Rating: 4.9673023 out of 5

Keywords: Kutz, Nathan Kutz, ICA, Independent Component Analysis

Id: _e4SN4TWlgY

Length: 50min 51sec (3051 seconds)

Published: Mon Dec 31 2018