MLAT Estimation Theory
52 flashcards · Computer Science · by Mathias777-crypto
What is estimated in Estimation Theory? How are the estimated parameters denoted? What do we always assume?
The parameters given data. ω^\hat{\omega}ω^ A distribution.
What's the way called to estimate parameters of an assumed distribution? What's done once a parameter like the mean of a Gaussian is estimated?
Estimation Theory It's plugged into a Gaussian distribution which then should reflect the data.
What is an estimator? What's it denoted mathematically?
It's a function to get an estimated parameter from given data points. ω^=g(x1,x2,...,xn)\hat{\omega} = g(x_1, x_2, ... , x_n)ω^=g(x1,x2,...,xn)
Of what nature is the estimated parameter ω^\hat{\omega}ω^ ?
It's a random variable, because the values it is generated from by an estimator are random variables, too.
Give an example of an estimator!
The arithmetic mean is an estimator of mu.
What is meant by the expected value of an estimated parameter?
It is quasi the average guess of the parameter after several guesses. Since guesses stem from estimators that are defined mathematically, their expected value can be calculated.
What does it mean for an estimator to be biased or unbiased? Give an example for an unbiased estimator.
If an estimator (estimation function) is biased, it means that the expected value of its outcome, the estimated parameter, is unequal to a distributions true parameter. If it's unbiased, the expected value is the same. The sample mean is an unbiased estimator, since its expected value is equal to μ\muμ .
Are bias and variance of an estimated parameter vector vector- or scalar-valued?
Bias: vector-valued Variance: scalar-valued
How is the bias of an estimator defined mathematically?
b(ω^)=E(ω^)−ωb(\hat{\omega}) = \mathbb{E}(\hat{\omega}) - \omegab(ω^)=E(ω^)−ω
How is the variance of an estimator defined mathematically?
Var(ω^)=E((ω^−E(ω^))T(ω^−E(ω^)))Var(\hat{\omega}) = \mathbb{E} \left( \left( \hat{\omega} - \mathbb{E}(\hat{\omega}) \right)^T \left( \hat{\omega} - \mathbb{E}(\hat{\omega}) \right) \right)Var(ω^)=E((ω^−E(ω^))T(ω^−E(ω^)))
How is the mean squared error defined mathematically? What can it be decomposed into?
mse(ω^)=E((ω^−ω)T(ω^−ω))mse(\hat{\omega}) = \mathbb{E} \left( ( \hat{\omega} - \omega)^T ( \hat{\omega} - \omega) \right)mse(ω^)=E((ω^−ω)T(ω^−ω)) It can be decomposed into the variance plus the squared bias of the estimated parameter.
What does it mean, if the (squared) bias of an estimator μ^\hat{\mu}μ^ of a Gaussian distribution is μ2(λ−1)2\mu^2 ( \lambda - 1)^2μ2(λ−1)2 ?
If λ\lambdaλ is 1, the estimator is unbiased, otherwise not.
If an estimated parameter is obtained by averaging multiple estimations, how does that effect the bias and the variance of an unbiased and the bias of a biased estimator?
Unbiased estimator: The bias stays 0, but the variance is divided by the number of estimations. Biased estimator: The number of estimations does not influence the bias.
What is a biased estimator called, that yields the true parameter by a large number of samples? Give an example for such an estimator!
Asymptotically unbiased estimator. σ2^=1l∑i=1l(xi−μ^)2\hat{\sigma^2} = \frac{1}{l} \sum_{i=1}^l \left( x_i - \hat{\mu} \right)^2σ2^=l1∑i=1l(xi−μ^)2 in a Gaussian distribution
Give an example of a biased estimator!
12l∑i=1lxi\frac{1}{2 l} \sum_{i=1}^{l} x_i2l1∑i=1lxi E(ω^)=12ω\mathbb{E}(\hat{\omega}) = \frac{1}{2} \omegaE(ω^)=21ω
What has to be given, so that an estimator of a distribution's variance is unbiased? How would you write that condition mathematically?
The expected estimated variance must be equal to the distributions true variance. E(σ2^)=σ2\mathbb{E}(\hat{\sigma^2}) = \sigma^2E(σ2^)=σ2
How does one, in general, determine if a variance estimator is biased or not? Adduce the starting point of an example.
Calculate its expected value and compare it with the true variance. When calculating, the variance formula must be inserted first (later, a formula for the estimated mean can be used, too (which can be unbiased). E(σ2^)=E(1l∑i=1l(xi−μ^)2)\mathbb{E}(\hat{\sigma^2}) = \mathbb{E} \left( \frac{1}{l} \sum_{i=1}^l \left( x_i - \hat{\mu} \right)^2 \right)E(σ2^)=E(l1∑i=1l(xi−μ^)2)
What is, with a Gaussian distribution, the famous unbiased estimator for the variance? Is it asymptotically unbiased?
σ2^=11−l∑i=1l(xi−1l∑i=1lxi)2\hat{\sigma^2} = \frac{1}{1-l} \sum_{i=1}^l \left( x_i - \frac{1}{l} \sum_{i=1}^l x_i \right)^2σ2^=1−l1∑i=1l(xi−l1∑i=1lxi)2 No, it's generally unbiased.
In which cases is it crucial to use the unbiased variance estimator of a Gaussian distribution and why?
If you have few samples, because if you have, say, 1000 data points, it doesn't matter much if you divide by 1000 or by 999. But if you have 4 samples, it makes a huge difference, if you divide by 4 or 3 to get the estimated variance.
What's the unbiased variance estimator in a Gaussian distribution, if instead of the estimator μ^\hat{\mu}μ^ , one uses the true μ\muμ ? If it looks different from the case when μ^\hat{\mu}μ^ is used, how come?
1l∑i=1l(xi−μ)2\frac{1}{l} \sum_{i=1}^l \left( x_i - \mu \right)^2l1∑i=1l(xi−μ)2 The terms are divided by lll only, because by using the true parameter μ\muμ , no degree of freedom is used up already.
What's a possible quality criterion for an estimator? What's the goal with respect to this criterion?
The mean squared error (MSE) To minimize it.
What is a MVU? How is it motivated?
The Minimum variance unbiased estimator is an estimator that was modified such that its variance is minimized. It is important because in the search of an estimator with the minimum mean squared error, the bias term is set to 0 (because the true parameter ω\omegaω is unknown), which leaves the variance term, that must be minimized.
Does a MVU always exist? Give a graphical example for when a MVU does not exist! Also give a graphical example for when it does exist!
No.
What's a common effect of a large dataset on the variance of an estimator?
The variance decreases.
What does one need to find the MVU? Why?
The Cramér-Rao lower bound (CRLB). Because if an estimated parameter's variance reaches the CRLB (for all parameter values), we can be sure that it's the MVU.
What is a nice way to obtain an estimator?
Maximum likelihood estimation (MLE)
Can a biased estimator be unbiased?
Yes, in an asymptotic way.
What's the formula for the likelihood L\mathcal{L}L ?
L(X;ω)=∏i=1lp(xi;ω)\mathcal{L} (X; \omega) = \prod_{i=1}^l p(x_i; \omega)L(X;ω)=∏i=1lp(xi;ω)
How to construct a MVU if it either doesn't exist or it's unknown? Which properties should it have?
By MLE an estimator that approximates an MVU can be found. It should be asymptotically efficient and asymptotically unbiased.
How can the likelihood be described informally?
It's the probability that a model reproduces a dataset.
What are two common tricks to calculate the likelihood? How does that look like expressed by mathematical statements?
Instead, the negative log-likelihood LLL is calculated. On the one hand the log, because it changes the product into a sum, which is easier to compute, while not affecting the location of the maximum. On the other hand the minus, because minimizing is more usual than maximizing. Therefor, the location of the minimum is sought. L(X;ω)=∏i=1lp(xi;ω)∣ln()\mathcal L (X; \omega) = \prod_{i=1}^l p(x_i; \omega) \vert \ln ()L(X;ω)=∏i=1lp(xi;ω)∣ln() lnL(X;ω)=ln(∏i=1lp(xi;ω))\ln \mathcal L (X; \omega…
How is the minimum of the negative log-likelihood function found? How to represent that mathematically?
The minimum can be found by differentiating with respect to the parameter that shall be estimated. Then, the first derivative is set to 0. The second derivative at that point must be > 0. The resulting location is the estimated parameter by MLE. ω^=argminω(L)\hat{\omega} = argmin_{\omega} \left( L \right)ω^=argminω(L)
What's the estimator for the mean value of a Gaussian distribution obtained by Maximum likelihood estimation?
1l∑i=1lxi\frac{1}{l} \sum_{i=1}^l x_il1∑i=1lxi
Is any estimator obtained from maximum likelihood estimation unbiased? If no, give an example!
No. The estimator for the variance of a Gaussian distribution derived from MLE is biased, however, asymptotically unbiased.
What's the important formula for the Fisher Information (the one valid if all regularity conditions are met)?
IF(θ)=−EX[∂2∂θ2lnp(x;θ)]I_F(\theta) = - \mathbb{E}_X \left[ \frac{\partial^2}{\partial \theta^2} \ln p(x; \theta) \right]IF(θ)=−EX[∂θ2∂2lnp(x;θ)]
What are, informally, the regularity conditions of the Fisher Information matrix?
The expected score must be 0.
What does the Fisher Information tell?
It tells the average curvature of the negative log likelihood of one or more samples! The negative log likelihood is taken, which results in a function that represents the distribution of parameter values with some being the most probable. This function is differentiated with respect to the desired parameter which results in another function that crosses the value 0 at some location, which is the desired estimated parameter. If one further differentiates the first derivative, the slope of the f…
What does the curvature of the second derivative of the negative log-likelihood typically correspond to?
To the curvature of the data distribution itself.
What else does the Fisher Information tell since one can calculate the average curvature of the negative log-likelihood of one or more samples?
It additionally tells, in comparing its value for different amounts of samples, how much the amount of samples matters for finding a nice estimator. This varies from type to type of distribution.
The higher the Fisher Information, the more information do some samples carry about an estimated parameter.
Is the Fisher Information taken of a distribution and one of its parameters or of an estimator?
Distribution and parameter
How to calculate the Cramér-Rao Lower Bound? Is it a lower or an upper bound? What is it the bound for? What's the mathematical statement that sums all that information up? What does it mean?
1IF(ω)\frac{1}{I_F(\omega)}IF(ω)1 Lower. A bound for the covariance of estimated parameters. Cov(ω^)≤1IF(ω)Cov(\hat{\omega}) \leq \frac{1}{I_F(\omega)}Cov(ω^)≤IF(ω)1 It means that the (co-) variance of an estimated parameter (which we want to keep as low as possible) is at least the inverse of the Fisher Information of the parameter of a distribution.
Do the estimators for the mean and the variance of a normal distribution influence each other? What's the consequence?
No. Their covariance values are 0.
What's the general plan when handling the CRLB?
Both the covariance matrix of the estimated parameters and the Inverse of the Fisher Information of the parameters of the distribution must be calculated and compared. If they are equal, the MVU is retrieved.
When is an estimator efficient?
If it reaches the CRLB.
Does a MVU always reach the CRLB? What does that mean concerning efficiency?
No. A MVU is not always efficient.
What's the χ2\chi^2χ2 -distribution? Why is it adduced here? What is σ2\sigma^2σ2 and kkk in this derivation of σ2^\hat{\sigma^2}σ2^ ?
It's the probability distribution of the sum of k squared outcomes drawn from a standard normal distribution. Because the estimated variance of a normal distribution can be modelled by it. σ2\sigma^2σ2 : 2(k−1)2 (k - 1)2(k−1) kkk : μ\muμ
What's the starting point to calculate the variance of the variance estimator of a normal distribution? How do you proceed?
(l−1)σ2^σ2∼χl−12∣Var()\frac{(l - 1) \hat{\sigma^2}}{\sigma^2} \sim \chi_{l-1}^2 \vert Var()σ2(l−1)σ2^∼χl−12∣Var() Var((l−1)σ2^σ2)=2(l−1)Var \left( \frac{(l - 1) \hat{\sigma^2}}{\sigma^2} \right) = 2 (l - 1)Var(σ2(l−1)σ2^)=2(l−1) (l−1)2σ4Var(σ2^)=2(l−1)\frac{(l - 1)^2}{\sigma^4} Var(\hat{\sigma^2}) = 2 (l - 1)σ4(l−1)2Var(σ2^)=2(l−1) Var(σ2^)=2σ4l−1Var(\hat{\sigma^2}) = \frac{2 \sigma^4}{l-1}Var(σ2^)=l−12σ4
How can you invert the Fisher Information matrix?
Simply calculate the inverse of the matrix elements.
What does it mean if an estimator is asymptotically efficient? Give an example of the principle!
It means that the variance of the estimator approximates the CRLB with a large amount of samples.
What does MLE usually yield with large datasets? The MLE is asymptotically efficient and asymptotically unbiased (sometimes unbiased anyway). How can these properties be expressed mathematically?
The MVU. liml→∞Cov(ω^)=CRLB\lim_{l \rightarrow \infty} Cov(\hat{\omega}) = \text{CRLB}liml→∞Cov(ω^)=CRLB liml→∞Ep(x;w)(ω^)=ω\lim_{l \rightarrow \infty} \mathbb{E}_{p(x;w)} (\hat{\omega}) = \omegaliml→∞Ep(x;w)(ω^)=ω
What further property does MLE have (Hint: invariant under parameter change)? What does that mean?
If a distribution's parameter is changed by a function ggg to a new parameter, the ML estimator can be obtained by applying ggg to the new estimator.