Pillai: Examples on Cramer-Rao Bound

so if theta theta is an unknown parameter and theta hat is an unbiased estimator then the chimera bone says that the variance of theta hat is going to be greater than or equal to one over expected value of D log of D theta squared or you can also write it as minus one or the second derivative of D log of so let me do a couple of examples where I'll blow couple of cases where it's an efficient estimator then I'll show you a case where it is not an efficient estimator so that is our classic example so let's assume that the data is Gaussian with mean and variance and here you will do one by one we'll assume that the mean is unknown so the joint density function is of course the data is independent so it's the product of the density functions I write this mu here just to show that the density function of course is a function of MU also so the as you know this will denote a 1 over 2 pi sigma squared n by 2 eat both - super ok so that's the way it will turn out to be and we can compute the Camaro remember look at here to compute the Camaro bond you don't need a unbiased estimator so let's just compute the Camaro bound first so we need to take the logarithm of this logarithm of f of X comma theta is so from here minus n by 2 log as above 2 Pi Sigma squared a minus Sigma X I minus mu whole squared over 2 Sigma square so what is it oh yeah yeah say theta here right I mean mu here of course the parameter is mu right is that what you're saying yeah so let's take the derivative so D log F comma mu with respect to MU so there is no mu here so if you look at this derivative of this quantity is 2 multiplied by these 2 2 cancels so you get Sigma X I minus mu this other two comes here cancels with this over Sigma squared are you good then multiplied by the derivative of this quantity which is minus 1 so minus minus cancels you get this expression you could sit here if you want so remember we have two expressions so we can either square this I'll do it both ways let me take the second derivative one more second derivative also so I'll show you both the ways here so this is going to be so second derivative D mu squared so derivative with respect to MU so there are two expressions here this is n times mu so this is N over Sigma squared right so now of course if you also if you take the expected value of this that's the same quantity so there's nothing so we already have this expression expected value of this is this a constant so that itself here you have to square this expression so if you Square D log F X comma theta naught theta D mu you get square XY minus mu squared or Sigma foo now you have to take the expectation here so this is a little more work so this is going to be so if you use the second expression we already have the answer because look at here so the answer here is the this cube the Sigma square to see or to be minus 1 over expected value of the second derivative of F with respect to MU squared so that's going to be Sigma squared over n inverse of this right so that's the Camaro bound yes now I just for your sake I'll do this way also we will see now you have to take the expected value of this but then here I equal to 1 since choir so this is double summation I hear J expected value of x I minus mu x xj minus mu so we have God divided by Sigma for now this you have to consider two cases i equal to j i is not equal to j right so that's going to be expected I II quality J will be expected value of x i minus mu to the power 4 right I equal to 1 through n because squared so this is going to be square right over Sigma fool and here what do you get here we have the summation double summation expected value X I minus mu x xj minus i I not equal to J so what's the what is the answer on the second node anybody that's 0 because when I is not equal to J they are independent so X each of this is so this is expected value multiplied by this and each is 0 here this is Sigma squared but n times Sigma squared so here Sigma squared cancels you get n over Sigma squared is the same as here so the answer would turn out to be you know when you flip it you get Sigma squared over you don't have to do both the ways I just did it because it's refers the problem sometimes so one is easier than the other most of the time this is easier now let's say let's say because we know what everybody remembers from here what is an efficient what is an estimate unbiased estimator in the case of if the data is Gaussian what is an unbiased estimator for the mean anyone all right so let's see what happens to sample mean sample mean is 1 over n Sigma X I I equal to 1 through n now you know that this is linear combination of Gaussian so this is Gaussian what is the mean of this random variable anybody Oh mean is mu because mu / and so mean is V and what is the variance anybody weigh all of them are independent so it's the sum of the variance divided by N squared so M Sigma squared over N squared so that Sigma squared over it so look here we have ik in this case we got lucky because we the variance agrees with the crab a rowboat so this is an efficient estimator and then we know that this is also the mlst beta of course you can do that quickly because what is ml estimator you take the logarithm look at here you take the logarithm and take the log of the van take the derivative which is here so to find the male estimator you equate this to 0 this is equal to ax equal to theta hat ml so if you equate this to 0 from here you can see that from here you can see n bo hat is Sigma X I so this gives you n mu hat ml is it my exile or you hit mu hat x ml is 1 over and Sigma X I I equal to 1 but that's your X bar so X bar is also the maximum likelihood estimator as the theorem says so this is your this sample mean is efficient and it's also the maximum and consequently is the maximum likelihood estimator so since we have a Gaussian case suppose your parameter is Sigma square design No okay so I'll take the case mu is known as the second example mu is known Sigma square DS are known so let me write the density function in terms of Sigma squared so I'm so Sigma squared I'm going to call it theta so the density function is what somewhere here so in so Sigma squared I'm going to call it theta so that's going to be or in fact I'll just directly go to logarithm of that so the logarithm of the density function is it's right there right n by 2 minus n by 2 log to point theta minus X I minus mu mu 1 squared over 2 theta right remember Sigma squared is our node so I'm going to call Sigma to be Sigma squared to be theta so it's the same problem except I flipped now here I am assuming that this is in the first four case I assumed this to be known what we did did because everywhere Sigma squared is there now I am assuming the other way mu is known Sigma squared is unknown so we should take the derivative with respect to the unknown yeah there is a summation right if you do the derivative this is going to be minus n by 2 theta log 2 pi plus log theta 1 over theta right or - so what is the derivative this is just a constant X I minus mu the whole square is a constant derivative of 1 over theta is what minus 1 over theta squared is so this is equal to 0 so you can see the theta hat turns out to be so you actually have a maximum likelihood estimator here but we don't have this is the first derivative so let me I can do it this or I can take one more derivative or we can just do it let's do one more there so d squared the log f of X comma theta D theta squared otherwise you have to square it and take ad take care of the terms so that's going to be what is it anybody so this will become theta squared right yes - will go away here - will appear what is it - 2 Sigma X I minus mu the whole square I equal to 1 through n minus 2 to 2 cancels will be theta cubed so let's take that out the expected value you get like one more line if you want expected value of this is fine constant expected value of this is what anybody expected value of xn minus mu definition 0 I mean square there is a square here this is the variance Sigma school or Theta this is Theta the expected value goes inside so this is n n theta everybody agrees so this quantity because this expected value if you put it here and this is theta which is Sigma squared theta but Six Sigma equal to one is n theta theta theta cancels so this is going to be minus in theta squared right the attack will cancel with theta theta squared theta is Sigma squared so this is Sigma 4 so then you get this to be putting it so you get this to be the answer so the crumb arrow bound in this case turns out to be the inverse of that so that is 2 Sigma 4 and let's say I came up with this as an estimator this is an estimator differently because it is a function of data right remember mu is known excise of the data so everything is known here so what is the expected value of this anybody expected value on the right side Sigma's good so this is an unbiased estimator so I got lucky so this is a classic example so this is an unbiased estimator for theta and let's find it so variance so how will you find the variance we go through the same way so you take the you know it's mean so you can find its square expected value of the square and subtract the mean for right so let's square it so squared this will be 1 over N squared and you have to square this quantity yes number variance is going to be expected value of theta hat squared minus Sigma 4 right because pyrrha square of the variance standard formula so I'm going to do this you can follow me this is one over in squid from here I'm going to square this outside there's already a square here on each term so then little bit double summation s term like this then a term like this J so this will be summation of X I minus mu the whole square sum on I summation of j X I minus mu the whole squared summation on J then you need to do the expected value here minus Sigma for so again we go through what we went through before so I've been i equal to j you get X summation expected value of x I minus mu to the fourth power write this over in squealed okay you put a equal to j i not equal to j so i goes from 1 through n and i not equal to j this will be expect i not equal to j right expected value of x i minus mu the whole square expected value of XJ minus mu the whole squared or n square minus Sigma 4 so let's put put each term so this term is look at this is Sigma squared this is Sigma squared right and there will be how many such terms remember total N squared terms n terms went away here so the second term will be N squared minus n multiplied by Sigma 4 over N squared minus Sigma 4 and the first term is anybody remembers what is the fourth moment what is expected value of the fourth moment of a Gaussian random variable 3 Sigma 4 isn't it I think it is 3 Sigma 4 G's anybody yeah it is 3 Sigma 4 right so this quantity is each one of them is this is 3 Sigma 4 this is just a standard expression so this is 3 n Sigma food over N squared now you can put it together see look at here you can cancel this in your head because of the N squared n squared will give you so the Sigma 4 will go away here you will have Sigma 4 over N here you have 3 3 Sigma 4 over N so the answer will turn out to be 2 Sigma 4 over and people can do it correctly but look that agrees with the Cravero bound again so this is also an efficient estimator so these are both our classical results yeah the bound of course can depend on the parameter right in this case that's true the bound difference of the parameter and remember the theorem didn't say that won't happen right so but the so this is in the if the day so let me give this summarize the result if the data is Gaussian and independent and if they remember the Gaussian has two parameters mu and Sigma squared if you treat one is unknown at a time so mu is unknown Sigma squared is known the other case is mu mu is known Sigma squared is unknown then both of them are if at the standard mean and the standard the the classic are they this Express this may this is the standard variance estimator both of them are efficient and unbiased and efficient and consequently this is also the maximum likelihood estimator which you can see from here if you put this equal to 0 you get so if you put this equal to 0 then you get theta hat will turn out to be there that one itself so this will also turn out to be theta hat ml and let me if I let me just do one more example connected with this and in this case I'll do boy so on the data we have the date the para data is poised on random variables and the parameter lambda in the Boyzone is unknown of course the problem is X is our poison with some unknown and lambda is unknown and the data is independent so the joint density function of all the parameters is the product of the density function of excise remember the random variable is discrete so this is e to the power minus lambda lambda to the power X I in other words each of them will be of this form erased minus lambda lambda to the power X I X I factorial so this is e raised to minus lambda lambda to the power C by X I so interesting he's the chimera mode you can find out even without knowing or wine unbiased estimator it just type so let me take the logarithm is going to be minus in lambda plus Sigma X I log lambda minus logarithm of the product of X I factor anyway this is uninteresting because it's not a function of lambda and we are going to take the derivative with respect to lambda so I need to take the derivative with respect to lambda of this expression that's only the first two terms so you get minus N and then you get Sigma X I I equal to 1 to n oh this is the first derivative so you can square this or you can do the second derivative so if I do the second derivative of f of X comma lambda so that will be as you can see it be - Sigma X I I equal to 1 through n over lambda squared Y so to complete the Camaro bond we'll do the expected value of the second derivative that will be the expected value of this quantity what is the expected value of x I if X is our Poisson lambda so I equal to 1 through n so the numerator is - n lambda over lambda squealed so that's minus n over so this gives us the crime arrow bound to be minus 1 over the second derivative equal to let's say I have this is that see that's the Camaro bound now if you have an unbiased estimator we can check what is its variance so let's again play the sample mean and see what happens in this case so the sample mean will be X bar is 1 over and Sigma X I I equal to 1 so what's the expected value of X bar anybody expected value of this expected value of exercise R lambda n lambda oh so this turns out to be lambda so in lambda N equals what's unbiased estimator so let's see what is its variance so that's going to be the second moment minus the mean squared right so what's the second moment anyone so again they go through this so this expression is 1 over N squared expected value of Sigma X I Y whole square so to complete this use so you can write this as expected value of double summation this is on I so I X I Sigma X T so you do have to consider two cases again 1 over N squared minus lambda squared so when i equal to j this becomes summation expected value of x i squared and when i is not equal to j if this will be simply i not equal to j they are independent so this is expected value of x i expected value of XT minus lambda squared but here divided by N squared right so everybody remembers the expressions expected value of x is for those you haven't the expected value of X is lambda expected value of X point is how much any one lambda plus lambda because the variance is lambda right so each of this is lambda plus lambda squared so lambda but there is n of them so lambda plus lambda squared over n because in squid and when you come to here each of this is lambda so here lambda is lambda so that's lambda squared but even though it - one of them so NN cancels so plus n minus 1 lambda squared over n minus lambda squared so what do we get so we get [Music] yeah so you get the first expression and all over and then you get lambda square over and then here you get plus lambda squared right then you get minus lambda squared or n minus lambda squared so this cancels with this this cancels with this lambda over N so the answer is lambda over it but that looks over here that's the same as the femoral bone indicating again that this is an efficient estimator let me previously I showed you efficient estimators so let's again take data to be Gaussian with the MU and Sigma squared and the data is of course independent but for whatever reason the parameter of interest to me is mu squeal not mu of course if the parent remember we already did mu if the just to be if you say that your parameter of interest is mu then we know that X bar is efficient X bar which is the sample mean is an efficient estimator right because the Kamarov bound for MU was what was a sigma squared over we did this and this was also the variance of X bar we went through all this so this is an this is an efficient estimator for the parameter view so you'll say it so come now we are interested in new squid so the common sense answer if you know form you just square it that doesn't work as you we will see because if you simply score this it may no it won't be even an efficient estimator for whatever we are looking forward to it so let me complete this problem when you try to find an official estimator and then find its variance and then we will also find the Crimea outbound and see whether we agree or also premiere about VCC so let me so because we had this earlier and remember theta is mu squared that means mu is a square root of theta so the joint density function if you remember it was 1 over 2 Pi Sigma squared n by 2 e to the power minus Sigma X I minus mu but mu is theta right so theta is what we are interested in so this is the joint density function so let me take the logarithm of this X I minus mu y ou is square root of theta over 2 Sigma square so let me take the derivative of this twice with respect to theta there is no derivative here this 1 to 2 cancels minus goes away so Sigma X I minus square root of theta and the derivative of this with respect to theta so what is it 102 - also right yes so this of course you can write it as Sigma X I I equal to 1 through n no super square the square root of theta minus 1 over 2 Sigma square so I'm going to do the second derivative so this gives me so what's the derivative of this X to the power minus 1/2 would be what theta to the power of minus 1 is minus F - half-price so that would be - food but yes why not I mean remember this is theta to the power half in the denominator that is Theta to the core - f so it's derivative is minus five x remember theta is what theta was there with a taste mu squared so this of course now I can write it as minus Sigma X I over for write theta theta is mu squared so that this mu cubed so let me take the expected value here so that's the expected value here expected value here expected value of x is what anybody you know no expected value of x I just move right so mu mu cancels so you get this to be minus 1 over 4 Sigma squared mu square there's also an N here right so this gives you the criminal bond for MU squared is minus 1 over expected value of b squared log D theta squared this there's another way to find this also come in a mat so this is one all that for Sigma squared mu squared over okay so here is again who made that comment here again the parameter is in the bound this is only this is only part of the problem we just found the crummy remote now the interesting the difficulty person is always give me an unbiased estimator and then give me an unbiased estimator which is agrees with this if you can find out which is an efficient estimator so remember X bar is good for MU so if you say oh yeah then I might not try this species unfortunately this is not true that's the problem in that's the problem in the probability and statistics just because you know the if you square the random variable you are not going to square the mean is not going to get split so what is the square of X bar and what is the mean of this one but in this case it is easy X bar is normal with mean mu and variance Sigma squared over N so expected value of x square x bar is x-bar squared is what that's a low standard theorem is going to be variances you can you can do it chi-squared what the variance if you want well but first of all chi-square this is a nonzero mean to me but also we can clear it so expected value of x is also we get this is you know the answer to this is EC in this case it happens to be easy so you see again moral of this story because the random variable is unbiased it doesn't mean if you square it is going to be unbiased it is not unbiased but in this case it is easy to create an unbiased random variable because if I do this so I am assuming obviously at this case Sigma squared is known right so we have an unbiased estimator for for so this is an unbiased estimator for mu hat I mean is our voice for not view hat is unbiased for useful so the last thing that remains is find the variance of this unbiased estimator so variance of theta hat equals variance is what expected value of theta hat squared minus its mean squared so this is what I'm going to call theta hat minus means quid mean expected value of theta hat squealed that's a standard expression for the variance so the first one we don't know we will find out expected value of theta hat squint - mean we found out look at your various new squared so - before so this is expected value of x x-bar squared minus sigma squared over n squared right so minus mu fool so this is going to be so there are three four terms x-bar fourth mean this fourth moment of x-bar minus 2 Sigma squared and expected value of x bar speed and the third term is Sigma 4 over N squared plus Sigma 4 over n squared minus mu 4 so this is a standard Gaussian result if X is Gaussian with mean mu and variance Sigma squared expected value of x is mu this is mu squared plus Sigma squared expected value of x cubed is mu cubed plus 3 Sigma 4 and expected value of x 4 is mu 4 plus 6 mu squared Sigma squared three Mu Sigma squared doesn't matter here but this is gonna be easy so we only need the fourth one except that in our case remember we are dealing with the X bar so we get here expected value of x 4 bar this is what we want here from there it will be it's a fourth mean plus 6 Mu Sigma squared or n plus 3 Sigma 4 over N squared so let me substitute that and see what happens so that's going to be mu 4 plus 6 Mu Sigma squared over n plus 3 Sigma 4 over N squared minus 2 Sigma 4 over in look at here expected value of x 4 bar is 1 so this is 2 Sigma squared then X X bar squared X bar squared is here right so that's going to be mu squared plus Sigma squared over n plus Sigma 4 over n minus 4 so view 4 cancels with service and something else will cancel also what care what else cancels what this is great yeah that cancels yeah here it cancels so that gives you right now it gives you 2 Sigma 4 so this goes with this this toy took care of it then what yeah 2 Sigma 2 Sigma 4 over and so that cancels to what ok speak up loud what oh yeah this is 3 plus 1 4 minus 2 so it'll be 2 so that's good ok so we took care of this also right then what is remaining you have 2 Sigma squared mu squared over N hey there is a new school this is nice wait let me let me double check right you all right that was mutilated so where is this also this is what you're saying is it so six view squared will cancel with two min squared so we'll get four right I remember that summer subtype is 4 mu squared Sigma squared over so this is the actual medians where is the chimera boat look at there the Camaro hold is only this term so the variance is higher than the Cravero bomb so here is a case look at the calculations involved but here is a case the estimator is not efficient but they're interesting questions before I finish this in this case is can I find another estimator which has got even lower variance because this possibility is over remember in this case the variance of your estimator is strictly greater not equal to the khmer romvong right this is not equal to this Sigma squared this year because if a squared CR is only for MU squared Sigma squared over N where as the variance has got this extra toe so the question is can you reduce can you find another estimator with lower variance this we will address in a class of something like possible so this is an estimator which is not efficient so I give you examples of official estimators not efficient estimators

Channel: Probability, Stochastic Processes - Random Videos

Views: 4,731

Rating: 4.9333334 out of 5

Keywords:

Id: PpcLCnwBFe0

Length: 46min 35sec (2795 seconds)

Published: Fri Feb 22 2019