STAT 4352 - Mathematical Statistics w/ Efromovich Spring 22'
QUIZ 1: Q4 Let X₁, X₂,...Xₙ ~ Expon(θ). Show that is the minimal variance unbiased estimator.
"minimal variance unbiased estimator." = CRLB CRLB: Var[x̄] ≥ 1/(n*F(i)) where F(i) = E[(d(ln(f(X)))/dθ)²] f(X) = θ⁻¹exp(-x/θ) ln(f(X)) = -x/θ - ln(θ) d(ln(f(X)))/dθ = x/(θ²) - 1/θ (x/(θ²) - 1/θ)² = x²/θ⁴ - 2x/θ³ + 1/θ²; E[x²/θ⁴ - 2x/θ³ + 1/θ²]; where it is given that E[x] = θ, Var[X] = θ² (from exponential function identities) :: E[X²] = Var[X] + (E[X])² by definition of variance; = θ² + (θ)² = 2θ². Now: E[x²/θ⁴ - 2x/θ³ + 1/θ²] = = E[x²/θ⁴] +E[-2x/θ³] + E[1/θ²] = E[x²]/θ⁴ -2E[x]/θ³ + 1/θ²E[] = 2θ²/θ⁴ - 2θ/θ³ + 1/θ² = 2/θ² - 2/θ² + 1/θ² = 1/θ² F(i) = 1/θ² Var[x̄] ≥ 1/(n*F(i)) = Var[x̄] ≥ 1/(n*1/θ²) = Var[x̄] ≥ θ²/n Now, solve for Var[x̄]: Var[n⁻¹ΣXi] = n⁻²Var[ΣXi] = n⁻²ΣVar[Xi] = n⁻²(n*θ²) = θ²/n Var[x̄] = CRLB; meaning x̄ is the UMVUE.
Steps to Establish an Efficient Estimator
1) Check that the estimator is unbiased. 2) Calculate the variance of the unbiased estimator (where variance = MSE = E[(θ^ - θ)²] 3) Calculate the Fisher Information 4) Plug-in the Fisher Information in the right-side of CRLB and calculate the lower bound. 5) If the calculated variance = to the lower bound for all θ ∈ A, The estimator is efficient. Note: This must hold for all considered θ ∈ A; Otherwise it is not minimum variance and by definition not efficient.
Steps for finding MLE
1) Write down the joint density or joint pmf as a function in θ, namely: L(θ) = fX1,X2,..Xn(X1,...Xn|θ) = ΠfX(Xj|θ), θ ∈ A Where L(θ) is the 'likelihood function' 2) Find value θ^ₘₗₑ which maximizes the likelihood L(θ) over θ ∈ A. This value is known as the 'maximum likelihood estimator'. θ^ₘₗₑ = (arg(max))ΠfX(Xj|θ) IMPORTANT NOTE FOR STEP 2: If θ defines the support of fX(x|θ), and this occurs if θ is in the indicator function: You find the MLE via a graphic of L(θ); L(θ) is not differentiable w.r.t the parameter at hand. If θ does NOT define the support, then θ is not in the indicator function: MLE can be found via differentiation of L(θ), finding extreme points, and then an extreme point with largest value of L(θ) is the MLE. It may also be more convenient to work with the log-likelihood function ln(L(θ)) in steps 1 and 2.
Consistent Estimator (Consistency)
A consistent estimator θ^ of θ becomes closer and closer to θ as sample size n increases. Consistency is an asymptotic property. An estimator θ^ of parameter θ ∈ A is called consistent if and only if: θ^ converges to θ in probability, for any ε > 0 and θ ∈ A; typically established with the Chebyshev Inequality: P(|θ^ - θ| > ε) → 0 as n → ∞
Unbiased Estimator
A desirable property of x̄ (or x-) (an estimator) Let X be a random variable with distribution depending on parameter θ, and it is known that θ ∈ A, where A is a known set of possible values of the parameter. Let X1, X2, ... Xn be a sample from X and θ^ = θ^(X1,X2, ... Xn) An estimator is unbiased if and only if: for all values θ ∈ A, E[θ^] = θ
Bayesian Estimation
A process where we assume that the parameter of interest θ is a random variable with known so-called 'prior' distribution. In our cases we will deal with continuous random variables which is defined by a known prior density h(θ). The Bayes estimator: θ^b = E[θ|X1,X2,...Xn] = Integral from -∞ to ∞ of: fθ|X1,X2,...Xn(θ|X1,X2,...Xn)dθ where fθ|X1,X2,...Xn(θ|X1,X2,...Xn) = h(θ)fX1,X2,...Xn|θ(x1,x2,...xn|θ) /fX1,X2,...Xn(x1,x2,...xn) ** NOTICE THE θ|Xi FLIPS ** also notice the denominator does not include |θ. where fX1,X2,...Xn(x1,x2,...xn) is the marginal density of observations, defined as: fX1,X2,...Xn(x1,x2,...xn) = Integral from -∞ to ∞ of h(θ)fX1,X2,...Xn|θ(x1,x2,...xn|θ)dθ. a.k.a: the denominator is the integral of the numerator. Whereas h(θ) is the prior; fθ|X1,X2,...Xn(θ|X1,X2,...Xn), the contents of the integral in the original bayes estimator equation, is known as the 'posterior'.
Robustness
An estimator is robust if changes in an underlying distribution do not dramatically affect its statistical properties. There is no rigorous definition of robustness; i.e.: x̄ is always unbiased estimator of the mean, and hence the estimator is robust with respect to an underlying distribution of the sample. :: No matter the distribution chosen, x̄ is still an unbiased estimator.
Biased Estimator
An estimator is unbiased if and only if: for all values θ ∈ A, E[θ^] = θ If above equation does not hold for at least one value of θ from the set A, then the estimator is called BIASED.
There is a sample size n from X ~ Gamma(α,β). Find MME estimator of parameters α,β
BECAUSE THERE ARE TWO PARAMETERS: WE KNOW WE MUST USE SYSTEM OF EQUATIONS. E[X] = n⁻¹ΣXi E[X²] = n⁻¹ΣXi² where E[X] and E[X²] of Gamma(α,β) are: E[X] = αβ and by definition of variance; Var[X] = E[X²] - (E[X])² E[X²] = Var[X] + (E[X])² E[X²] = αβ² + (αβ)² = (α²+α)β² = α(α+1)β² Plug in values: αβ = n⁻¹ΣXi α(α+1)β² = n⁻¹ΣXi² :: αβ = x̄ α(α+1)β² = n⁻¹ΣXi² β = x̄/α Rearrange α(α+1)β²: (α²+α)(x̄²/α²) = (1 + 1/α)x̄² :: = x̄² + x̄²/α = n⁻¹ΣXi² = x̄²/α = n⁻¹ΣXi² - x̄² α^ₘₘₑ = x̄²/(n⁻¹ΣXi² - x̄²) Now alternatively: α = x̄/β (α²+α)β² = ((x̄/β)² + x̄/β)β² = x̄² + x̄β = x̄² + x̄β = n⁻¹ΣXi² = x̄β = n⁻¹ΣXi² - x̄² β^ₘₘₑ = (n⁻¹ΣXi² - x̄²)/x̄
Bayes Example: Normal Random Variable Let a sample of size n be from X ~ Normal(µ,σ²) Find Bayesian Estimator
Bayesian approach is used when the parameter of interest µ is a realization of a random variable M with the prior Normal(µ₀,σ₀²) fM|X₁,X₂,...Xₙ(µ|x₁,x₂,..xₙ) = h(µ)fX₁,X₂,...Xₙ(x₁,x₂,...xₙ|µ)/fX₁,X₂,...Xₙ(x₁,x₂,...xₙ) = (2πσ₀²)⁻¹/²exp(-(µ-µ₀)²/2σ₀²) *Π(2πσ²)⁻¹/²exp(-(xj-µ)²/2σ²) /fX₁,X2,...Xn(x₁,x2,...xn) Note that the denominator does not contain µ. Take terms w.r.t µ: fM|X1,X2,...Xn(µ|x1,x2,..xn) ∝ exp(-µ²(nσ⁻² + σ₀⁻²)1/2 + µ(nx̄σ⁻² + ₀σ₀⁻²)) fY(y|a) = (2π(b²))⁻¹/²exp(-(y-a)²/2b²) ∝ exp(-y²/(2b²) + y(a/(b²))) µb = (σ²µ₀ + nσ₀²x̄)/(σ²+ nσ₀²) σ²b = (σ²σ₀²)/((σ²+ nσ₀²)) µ^b = (σ²µ₀ + nσ₀²x̄)/(σ²+ nσ₀²) and Variance of the Bayesian estimate is: V(µ^b) = σ²^b = (σ²σ₀²)/((σ²+ nσ₀²)) µ + nx̄₀σ⁻²
Consider a sample of size n from X ~ Normal(0,σ²). Find MME of σ².
Consider 2nd moment, because 1st moment is E[x] and is known to be 0. E[X²] = n⁻¹ΣXi² where Σ ranges from i=1 to n By definition of variance; Var[X] = E[X²] - (E[X])² where we know E[X] = 0; = Var[X] = E[X²] σ² = n⁻¹ΣXi² :: σ²^ₘₘₑ = n⁻¹ΣXi²
Consider Exponential(λ) distribution and a sample of size n. Show that x̄ is a sufficient statistic.
Consider the joint density. NOTE: Π. IS. PRODUCT. NOT. SIGMA. fX₁,X₂,...Xₙ(x₁,x₂,...xₙ|θ) = Π[exp(-xᵢ/λ)/λ] I (xᵢ > 0) where Π is from i=1 to n where exp(-xᵢ/λ)/λ is the density of an exponential distribution. = [exp(-Σxᵢ/λ)/(yⁿ)] [Π] I (xᵢ > 0) → divide by n to get x̄ as specified in the question = [exp(-n⁻¹Σxi/(λ/n))/(yⁿ)][Π] I (xᵢ > 0) = [exp(-x̄/(λ/n))/(yⁿ)][Π] I (xᵢ > 0) = [g(x̄,λ)][h(x₁,x₂,...xₙ)]
Sample Variance Derivation
Consider the variance identity: Σ(xi-x̄)² where Σ is from i=1 to n = Σ(xi²-2x̄xi+x̄²) = Σxi²-2x̄Σxi+Σx̄²) =Σxi² - 2nx̄² + nx̄² = Σxi² - nx̄² :: Now consider Σ(xi-x̄)².= E[Σ((xi-µ)-(x̄-µ))²] = E[(Σ(xi-µ)²)- n(x̄ - µ)²] = ΣE[(xi-µ)²] - nE[(x̄ - µ)²] = ΣVar[Xi] - nVar[x̄] plug-in Var[Xi] and Var[x̄] -> Var[Xi] = σ² -> Var[x̄] = σ²/n = nσ² - n(σ²/n) = nσ² - σ² = (n-1)σ² What can we multiply (n-1)σ² by to get σ²? 1/(n-1) :: Equivalently: = (n²/n - 1/n)Var[Xi] = ((n²-1)/n)Var[Xi] What can we multiply ((n²-1)/n)Var[Xi] by to get Var[Xi]? 1/(n-1) :: S² = Σ(xi-x̄)²/(n-1) = E[Σ((xi-µ)-(x̄-µ))²]/(n-1)
Sample Variance
DO NOT CONFUSE SAMPLE VARIANCE WITH VAR[x̄]. VAR[x̄] IS "VARIANCE OF THE SAMPLE MEAN" S² IS "SAMPLE VARIANCE" s² = Σ(xi-x̄)²/(n-1) where Σ is from i=1 to n
Sufficient Statistic Relationship
Definition of a Sufficient Statistic: Consider a sample X₁,...Xₙ from X w/ the density or pmf fX(x|θ) where θ is the parameter of interest. a statistic T = T(X₁,X₂,...Xₙ) is sufficient for parameter θ if and only if: The conditional joint density or conditional joint pmf satisfy: fX₁,X₂,...Xₙ|T(x₁,x₂,...xₙ|T,θ) = fX₁,X₂,...Xₙ|T(x₁,x₂,...xₙ|T) that is; the joint distribution of the sample does not depend on the parameter of interest.
Point Estimator
Denoted by ^, used to estimate a distribution. Point estimators have properties that make them better than other point estimators; such as: - Unbiased Estimator - Efficient Estimator - Consistent Estimator - Sufficient Estimator
Let X ~ Binomial(θ,n). Show that θ^ = (X+1)/(n+2) is an asymptotically unbiased estimator of θ.
E[(X + 1)/(n + 2)] = (E[X] + 1)/(n + 2) = (nθ + 1)/(n + 2) = θ(n/(n+2)) + 1/(n+2) = (n+2)θ(n/(n+2)) + (n+2)1/(n+2) = θn + 1 = θ + 1/n as n → ∞, E[(X + 1)/(n+2)] → θ (X + 1)/(n+2) is an asymptotically unbiased estimator of θ
EXTRA PROBLEMS 2: Let X ~ Poisson(λ). Find MME of λ²
E[X] = λ = x̄ E[X²] = Var[X] + (E[X])² = λ +(λ)² = λ+λ² = Σ(Xi)² λ² = Σ(Xi)² - λ λ² = Σ(Xi)² - x̄ λ²^ₘₘₑ = n⁻¹Σ(Xi)² - x̄
HW2 Q1: θ₁^ and θ₂^ are unbiased estimators of the same parameter θ. What condition must be imposed on the constants k₁ and k₂ so that k₁θ₁^ + k₂θ₂^ = θ
E[k₁θ₁^ + k₂θ₂^] = E[k₁θ₁^] + E[k₂θ₂^] = k₁E[θ₁^] + k₂E[θ₂^] = k₁θ + k₂θ = (k₁+k₂)θ, where parameter θ is the end target. thus, k₁ + k₂ must equal 1 for the conditions to be met.
HW2 Q2: Given a random sample of size n from a population that has a known mean µ and the finite variance σ², show that: n^-1Σ((xi - µ)²) is an unbiased estimator of σ².
E[n^-1Σ((xi - µ)²)] = n^-1E[Σ((xi - µ)²)] = n^-1ΣE[((xi - µ)²)] NOTE: E[((xi - µ)²)] is a known equation that equals Var[xi] = n^-1ΣVar[xi] = n/n * Var[xi] = Var[xi] = σ² thus, n^-1Σ((xi - µ)²) is an unbiased estimator of σ².
Let us have a sample of size n from X with µ = E[x] be the parameter of interest. Under what condition on a sequence a1, a2, ... of real numbers, is a statistic µ^ = ΣaiXi an unbiased estimator of µ?
E[µ^] = E[ΣaᵢXᵢ] = ΣaᵢE[Xᵢ] = Σaᵢ * µ = µΣaᵢ where Σ is from i=1 to n µ^ is an unbiased estimator when Σaᵢ = 1; or rather: the sum of a₁, a₂, ... = 1.
Let us have a sequence of independent RVs X₁, X₂, ... such that E[Xᵢ] = bₜµ where bₜ are known numbers and µ is the parameter of interest. Under what condition on a sequence a₁, a₂, ... of real numbers: Is a statistic µ^ = ΣaᵢXᵢ an unbiased estimator of µ?
E[µ^] = E[ΣaᵢXᵢ] = ΣaᵢE[Xᵢ] = Σaᵢbₜµ = Σaᵢbₜ*µ = µΣaᵢbₜ where Σ is from i=1 to n µ^ = ΣaᵢXᵢ is an unbiased estimator when Σaᵢbₜ = 1
Let X₁, X₂, ... Xₙ be a sample from X ~ Normal(0,σ²). Show that estimator σ²^ = n⁻¹ΣX² is an unbiased estimator of σ².
E[σ²^] = E[n⁻¹ΣX²] where Σ is from i=1 to n = n⁻¹E[ΣX²] = n⁻¹ΣE[X²] = n⁻¹ΣV(X) → because (E[X])² = 0 = n⁻¹(nσ²) = σ² n^-1ΣX² is an unbiased estimator of σ²
F distribution
F(v1, v2) = ((ΣX²)/v1) / ((ΣY²)/v2) where Σ is from i=1 to v1 and Σ is from i=1 to v2 respectively Used to analyze ratio of two sample variances from different populations
True or False, is a sufficient statistic unique/distinct?
FALSE i.e. x̄ and 2x̄ are both sufficient statistics because you know 2x̄ if you know x̄, and vice versa.
Consider a sample of size n from Bernoulli(θ). What is the sufficient statistic?
Factorization Theorem: fX₁, X₂, ... Xₙ(x₁, x₂, ... xₙ) = Π[(θ)^xᵢ(1-θ)^(1-xᵢ)] where Π is from i=1 to n = Π[(θ/(1-θ))^xᵢ*(1-θ)] = (θ/(1-θ))^Σxᵢ*(1-θ)ⁿ = [(θ/(1-θ))^Σxᵢ*(1-θ)ⁿ] * [1] = [g(Σxᵢ,θ)][h(x₁, x₂, ... xₙ)] Σxᵢ is a sufficient statistic of Bernoulli(θ). NOTE: x̄ is proved equivalently with one final step of dividing by n outside of Σ and multiplying by n inside. also NOTE: Σxᵢ ~ Binomial(θ,n)
QUIZ 1: Q9 Let X be Binom(n, θ) and θ is believed to be a realization of a RV Θ ∼ Beta(α, β). Find a Bayes estimator of θ.
Find posterior distribution: h(θ)f(x|θ)/fX(x) h(θ) = Γ(α+β)/(Γ(α)Γ(β))θᵃ⁻¹(1-θ)ᵇ⁻¹ I (0 < θ < 1) f(x|θ) = (n!/(x!(n-x)!)θˣ(1-θ)ⁿ⁻ˣ where fX(x) does not involve θ; h(θ)f(x|θ) ∝ θ⁽ᵃ⁺ˣ⁾⁻¹(1-θ)⁽ᵇ⁺ⁿ⁻ˣ⁾⁻¹ so we can conclude that: Posterior ~ Beta(α+x, β+n-x) Meaning; θ^b = E[Posterior] = (α+x)/(α+x+β+n-x) = (α+x)/(α+β+n)
There is a sample of size n from X ~ Uniform(α,1). Find MME of α
First moment approach: E[X] = (α+1)/2 (α+1)/2 = x̄ α^ₘₘₑ = 2x̄ - 1
Let X be Poisson(λ), with a sample size n available. Find MME of λ²
First moment approach: E[X] = λ Var[X] = λ λ² is not obtainable. Second moment approach: E[X²] = ? By definition of variance; Var[X] = E[X²] - (E[X])² E[X²] = Var[X] + (E[X])² E[X²] = λ + λ² Now; write system of equations: E[X] = x̄ E[X²] = n⁻¹ΣXi² Where Σ ranges from i=1 to n Plug in found values: λ = x̄ λ + λ² = n⁻¹ΣXi² Solve w.r.t λ²: λ² = n⁻¹ΣXi² - λ λ² = n⁻¹ΣXi² - x̄ λ²^ₘₘₑ = n⁻¹ΣXi² - x̄
QUIZ 1: Q2 Let a sample of size n from Unif(0, θ) be given. What is the pdf of the sufficient statistic?
For sufficiency: Use Factorization Theorem. = Π1/(θ-0)|(0< xᵢ < θ) = Π1/(θ)|(0< xᵢ < θ) = θ⁻ⁿΠI(0< xᵢ < θ) = θ⁻ⁿΠI(0< x₍₁₎ < x₍ₙ₎ < θ) X₍ₙ₎ is the sufficient statistic. for T = X₍ₙ₎, t ∈ [0,θ]: P(T ≤ t) = ΠP(Xᵢ ≤ t) = (t/θ)ⁿ meaning by definition of pdf given cdf: pdf = d(P(T ≤ t)/dt =d((t/θ)ⁿ)/dt = ntⁿ⁻¹θ⁻ⁿ I (0 < t < θ)
Prove that X/(n-1) is an asymptotically unbiased estimator of θ, θ ∈ [0,1]; given X ~ Binomial(θ,n).
For this question, use b(θ) bias equation to solve (note, for an estimator to be unbiased, b(θ) = 0, so negative numbers can be moved to the other side: b(θ) = E[X/(n-1)] - θ = E[X/n + X/(n(n-1))] - θ = n⁻¹E[X] + (n(n-1))⁻¹E[X] - θ = nθ/n + nθ/(n(n-1)) - θ = θ + θ/n-1 - θ = θ/n-1 Alternatively: E[X/(n-1)] - θ = (n-1)⁻¹E[X] - θ = θn/(n-1) - θ bring -θ into the fraction: = (θn - (θ(n-1))/n-1 = θ/n-1 As n → ∞, θ/(∞-1) → 0. X/(n-1) is asymptotically unbiased.
Chi-Square
IF X IS NORMAL(µ, σ²) S² = (1/(n-1))Σ(Xi - x̄)^2 has Chi-Square distribution with n-1 degrees of freedom. where Σ is from i=1 to n
Theorem 10.1
If S^2 is the variance of a random sample from an infinite population with the finite variance σ², Then E[S^2] = σ² Proof: E[S^2] = E[(n-1)^-1 * Σ(Xi - x̄)² where Σ is from i=1 to n = (n-1)⁻¹*E[Σ((Xi - µ) - (x̄ - µ))²] = (n-1)⁻¹*(Σ(E[(Xi-µ)²])-n*E[(x̄-µ)²]) E[(Xi - µ)² = σ²; E[(x̄-µ)²] = σ²/n = (n-1)⁻¹*(Σσ² - n(σ²/n)) = (n-1)⁻¹ * (nσ² - σ²) = σ² Proved.
Lowercase and Uppercase Notation
It is best practice to use x for numerical values, X for random variables. i.e. ΣXi where Σ is from i=1 to n X - random variable i - subscript # in the order of i = 1 to n(but I can't really do it cleanly with Quizlet)
Consider a sample of size n from X ~ Normal(µ,σ²), µ ∈ (-∞,∞) Find MLE for pair (µ,σ²).
L(µ,σ²) = Π((σ²2π)⁻¹/²exp(-(Xj - µ)²/(2σ²))) I (-∞ < Xj < ∞) = (σ²2π)⁻ⁿ/²exp(-Σ((Xj - µ)²/(2σ²))) l(µ,σ²) = (-n/2)ln(σ²2π) - Σ((Xj - µ)²/(2σ²)) now, take partial derivate w.r.t each variable and create system of equations. Note that two parameters are present so we'll need a system. dl(µ,σ²)/dµ = σ⁻²Σ(Xj - µ) dl(µ,σ²)/dσ² = -n/(2σ²) + 1/(2σ⁴)Σ((Xj - µ)² Set equal to 0: σ⁻²Σ(Xj - µ) = 0 -n/(2σ²) + 1/(2σ⁴)Σ((Xj - µ)² = 0 σ⁻²Σ(Xj - µ) = 0 = Σ(Xj - µ) = 0 = ΣXj - nµ = 0 = µ = ΣXj/n = x̄ -n/(2σ²) + 1/(2σ⁴)Σ((Xj - µ)² = 0 times 2σ²: -n + (1/σ²)Σ((Xj - µ)² = 0 = σ² = Σ((Xj - µ)²/n Where it is given from the first equation that: µ = ΣXj/n = x̄ = σ² = Σ((Xj - x̄)²/n (µ^ₘₗₑ,σ²^ₘₗₑ) = (x̄,Σ((Xj - x̄)²/n)
A sample of size n has RV X ~ Uniform(α,1), 0 ≤ α < 1. Find MLE.
L(α) = Π(1/(1-α)) = (1/(1-α)ⁿ) = (1-α)⁻ⁿ I (0 ≤ α ≤ Xⱼ < 1) Similarly; WE MUST GRAPH: We know that, at most: α = Xⱼ. Any other event, α will be less than Xⱼ. Thus we want the smallest available X; X₍₁₎. α^ₘₗₑ = X₍₁₎
QUIZ 1: Q7 Consider a sample of size n from Unif(α, β). Find the MLE of the pair (α, β).
L(α, β) = Π(β - α)⁻¹ = (β - α)⁻ⁿ I (α < Xi < β) = (β - α)⁻ⁿ I (α < X₍₁₎ ≤ X₍ₙ₎ < β) α, β are in the indicator function. A graphic is necessary. α^ₘₗₑ = X₍₁₎ β^ₘₗₑ = X₍ₙ₎
A sample of size n is from Uniform(0,β), β>0 Find MLE
L(β) = Π(1/(β-0)) I (0 < Xj < β) = Π(1/β) = β⁻ⁿ I (min Xj > 0) I (max Xj < β) NOTE::::::::: β IS IN THE SUPPORT. ONLY SOLVABLE BY GRAPHIC. Now, we know that: max Xj < β and min Xj > 0 So the graph will be 0 toward the start of the range, and will the maximum likelihood at whatever value is the maximum available; denoted by X₍ₙ₎. We know that β will be greater than any Xj possible, but we are nonetheless still aiming for β. β^ₘₗₑ = X₍ₙ₎
HW3 Q6: Use method of maximum likelihood to rework Q3; Where Q3 states: If X₁,X₂,...Xₙ constitute a random sample of size n from a population given by: g(x;θ) = { (1/θ)exp(-(x-δ)/θ) for x > δ, { 0 elsewhere Find estimators for δ and θ by the method of moments.
L(δ,θ): = Π((1/θ)exp(-(x-δ)/θ) | (x > δ) =θ⁻ⁿexp(-Σ((Xᵢ-δ)/θ)) | (X₍₁₎ > δ) ln(L(δ,θ) = -nln(θ) - Σ((Xᵢ-δ)/θ) Take derivative with respect to each variable: w.r.t θ, set to 0: -n/θ + (1/θ²)Σ(Xᵢ-δ) = 0 n/θ = (1/θ²)Σ(Xᵢ-δ) n = (1/θ)Σ(Xᵢ-δ) :: θ = n⁻¹Σ(Xᵢ-δ) w.r.t δ, set to 0: NOT POSSIBLE! δ is in the indicator function. The closest Xᵢ to δ is X₍₁₎. NOTE :: θ = n⁻¹Σ(Xᵢ-X₍₁₎) (δ^ₘₗₑ,θ^ₘₗₑ) = (X₍₁₎,n⁻¹Σ(Xᵢ-X₍₁₎))
QUIZ 1: Q8 Find the MLE of θ for a sample of size n from a normal population with mean θ and variance 1. It is given that θ ∈ Ω = [0,∞).
L(θ) = Π(2π)⁻¹/²exp(-1/2(Xi-θ)²) = (2π)⁻ⁿ/²exp(-1/2Σ(Xi-θ)²) ln(L(θ)) = -n/2ln(2π) - 1/2Σ(Xi-θ)² = -n/2ln(2π) - 1/2Σ(Xi² - 2Xiθ + θ²) = -n/2ln(2π) - 1/2Σ(Xi²)-1/2Σ-2Xiθ + -1/2Σθ²) = -n/2ln(2π)-1/2Σ(Xi²) +ΣXiθ -nθ²/2 f = -n/2ln(2π)-1/2Σ(Xi²) +nx̄θ -nθ²/2 d(f)/dθ = nx̄ -nθ set equal to 0: nθ = nx̄ θ = x̄ θ^ₘₗₑ = x̄
HW3 Q4: Use method of maximum likelihood to rework Q2; where Q2 states: Given a random sample of size n from a Poisson population, use the method of moments to obtain an estimator for the parameter λ
L(λ) = Π(λˣexp(-λ))/x! = λ^(Σxᵢ)exp(-nλ)/Σxᵢ! ln(L(λ)) = Σxᵢln(λ) - nλ - Σln(xᵢ!) now derivative w.r.t λ: = Σxᵢ/λ - n set equal to 0, solve for λ: λ = Σxᵢ/n λ = x̄ λ^ₘₗₑ = x̄
Cramer-Rao Lower Bound
Let X be a r.v. with density fˣ(x|θ) defined by parameter of interest θ; X₁, X₂, ... Xₙ is a sample of size n from X, θ ∈ A where the set A is given, and θ^ is an unbiased estimator of θ. for all θ ∈ A: V(θ^) ≥ 1/(n*Fisher Information) that is; variance of the estimator cannot be smaller than: 1/(nE[(d/dθ(ln(fX(X|θ)))²]) or in plain-text: 1 over n multiplied by the expectation of the squared value of the derivative of the natural log of the density (of our given r.v.)
Method of Moments Estimator (MME)
Let X1, X2, ... Xn be a sample from RV X, and the distribution of X is known up to a parameter θ; that is: we know fX(x|θ). Suppose that X has a finite kᵗʰ moment, that is E[Xᵏ] < ∞. E[Xᵏ] = n^-1ΣXiᵏ where Σ ranges from i=1 to n. Solve the equation w.r.t θ, and you'll have the Method of Moments Estimator, written as: θ^ₘₘₑ Note that θ is only present on the left side of the equation (in the E[Xᵏ]) and E[Xᵏ] is calculated traditionally. The ride side of the equation is the sample kᵗʰ moment. Typically either k=1 or 2 are used. If two parameters θ and v of the distribution are unknown, that is: we know fX(x|θ,v), then a system of equations is considered: E[Xᵏ¹] = n⁻¹ΣXiᵏ¹ E[Xᵏ²] = n⁻¹ΣXiᵏ² Where Σ is from i=1 to n. Solutions (θ^ₘₘₑ, v^ₘₘₑ) of the system is the method of moments estimator. k1 = 1 and k2 = 2 are the traditional choice.
Maximum Likelihood Estimator (MLE)
MLE is an estimator that is most of the time efficient. MLE is always a function of a sufficient statistic.
Prove that MSE = Var(θ^) given: θ^ is an unbiased estimator of θ.
MSE = E[(θ^ - θ)²] If an estimate is unbiased, it is known that E[θ^] - θ^ = 0. Var(θ^) = E[(θ^ - E[θ^])²] where E[θ^] = θ... = E[(θ^ - θ)²] Var(θ^) = MSE
Let θ^ be an estimate of parameter θ. Find the Mean Squared Error (MSE) of θ^.
MSE(θ^) = E[(θ^ - θ)²] = E[(θ^ - E[θ^] + E[θ^] - θ)}²] → Note that E[θ^] cancels; it is still the same equation (a+b)² = a² + b² + 2ab where a = θ^ - E[θ^] b = E[θ^] - θ = E[(θ^ - E[θ^])² + E[(E[θ^] - θ)²] + 2E(θ^ - E[θ^])(E[θ^] - θ) NOTICE: a² = E[(θ^ - E[θ^])² = Var[θ^] b² = E[(E[θ^] - θ)²] = (Bias[θ^])² and finally NOTE: E[θ^] - θ is a constant; i.e. E[ab] = bE[a] and E[θ^ - E[θ^]] = 0. 2ab = 2E[(θ^ - E[θ^])(E[θ^] - θ)] = 2(E[θ^] - θ)E[θ^ - E[θ^]] = 2(E[θ^] - θ)(0) = 0. Finally: MSE = Var(θ^) + (Bias(θ^))²
QUIZ 1: Q6 Consider a sample of size n from Unif(α, α + β). Find the method of moment estimator of the parameter g = 2α + β.
Method of Moments: E[X] = (α + α + β)/2 = (2α + β)/2 = α + β/2 Set first equation for M.o.M: α + β/2 = x̄ going back one step: (2α + β)/2 = x̄ = g/2 = x̄ g^ₘₘₑ = 2x̄
QUIZ 1: Q10 Let X be Normal(θ,σ²) and σ² be known. Find method of moments estimate of θ².
Method of Moments: E[X] = θ E[X²] = Var[X] + (E[X])² = σ² + θ² θ = x̄ σ² + θ² = n⁻¹Σ(Xi)² θ²^ₘₘₑ = n⁻¹Σ(Xi)² - σ²
HW2 Q8: Use Definition 10.2 to show that Y(1), the first order statistic: is a consistent estimator of the parameter α, of a uniform population with β = α+1.
NOTE!! Uniform Distribution is denoted as: Unif(α,β) meaning it is given that Y ~ Unif(α,α+1). Also note that because of this given information, we know for the chebyshev inequality that 0 < ε ≤ 1 only, any ε > 1 is not necessary to prove consistency of Y(1). Therefore, we must prove: lim(n→∞)P[|Y(1) - α| < ε] = 1, for 0 < ε ≤ 1. We also know: α ≤ Y(1) ≤ β by definition of order statistics, and thus: = α ≤ Y(1) ≤ α + 1 = 0 ≤ Y(1) - α ≤ 1 Therefore, we know Y(1) - α is guaranteed to land between 0 and 1. The absolute value part of the inequality is unnecessary. = lim(n→∞)P[Y(1) - α < ε] = 1, for 0 < ε ≤ 1. = lim(n→∞)P[α < Y(1) < α+ε] = 1, for 0 < ε ≤ 1. using Theorem 8.16: = Integral from α to α+ε [n*[1 - Y(1) + α]^(n-1)]dY(1) = change to (n*t^(n-1))(-dt), where -dt comes from: 1 - Y(1) + α = t -dY(1) = dt dY(1) = -dt and limits as: lower: 1 - Y(1) + α = 1 - α + α = 1 and upper: 1 - Y(1) + α = 1 - (α+ε) + α = 1-ε = Integral from 1 to 1-ε = [(n*t^(n-1))(-dt)] = n * Integral from 1 to 1-ε [t^(n-1) -dt] = -n * Integral from 1 to 1-ε [t^(n-1)dt] = -n((t^n)/(n)) = -t^n = -((1-ε)^n) -(1)^n) = 1^n - (1-ε)^n = 1 - (1-ε)^n = 1 as n→∞; Where (1-ε)^n →0 as n→∞; Proving under the original definition that Y(1) is a consistent estimator.
HW2 Q3: Use the results of Theorem 8.1 to show that x̄² is an asymptotically unbiased estimator of µ²
Note that Theorem 8.1 just states the expectation and variance of the sample mean; i.e. E[x̄] and Var[x̄]. ... That is: E[x̄] = µ Var[x̄] = σ²/n The problem: E[x̄²] using properties of variance: Var[x̄] = E[x̄²] - E[x̄]² -> E[x̄²]= Var[x̄] + E[x̄]² = σ²/n + µ² As n → ∞, σ²/n → 0, meaning as n → ∞, E[x̄²] = µ² thus, x̄² is an asymptotically unbiased estimator of µ².
Observe a sample of size n from X ~ Uniform(θ-1/2, θ+1/2), θ ∈ (-∞,∞). Find MLE of θ.
Observe that θ is in the range indicator, so we'll need to use a graphic. Start w/ range indicator; by definition of Uniform: θ-1/2 ≤ x ≤ θ+1/2 - 1/2: θ-1 ≤ x - 1/2 ≤ θ :: x - 1/2 ≤ θ from original range, + 1/2: θ ≤ x + 1/2 ≤ θ+1 :: θ ≤ x + 1/2 new range w.r.t x: x - 1/2 ≤ θ ≤ x + 1/2 a.k.a: Xi - 1/2 ≤ θ ≤ Xi + 1/2 where i=1 to n. For logic behind X₍₁₎ and X₍ₙ₎, we are looking for the tightest bounds possible; to make a more useful statistic. For that to happen, we need the biggest number available below θ, and the smallest number available above θ for the left and right bounds respectively. i.e. Xi - 1/2 ≤ θ :: X₍ₙ₎ - 1/2 > X₍₁₎ - 1/2, but both are valid. However, the difference between θ and X₍ₙ₎ - 1/2 is smaller, meaning it will be a tighter bound. θ ≤ Xi + 1/2 :: X₍ₙ₎ + 1/2 > X₍₁₎ + 1/2, for this same exact reason as above, we want the closest number possible to our estimator. Therefore, our new bounds are: X₍ₙ₎ - 1/2 ≤ θ ≤ X₍₁₎ + 1/2 X₍ₙ₎ - 1/2 ≤ θ^ₘₗₑ ≤ X₍₁₎ + 1/2
Bayes Example: Binomial Random Variable There is X ~ Binomial(θ,n), Bayes approach with θ being realization of a Beta(α,β) r.v. Θ. (a.k.a: the prior is a beta distribution) Find the Bayesian Estimator.
PROFESSORS METHOD: If Θ ~ Beta(α,β): h(θ) = Γ(α+β)/(Γ(α)Γ(β))θᵃ⁻¹(1-θ)ᵇ⁻¹ I (0 < θ < 1) E[Θ] = α/(α+β) Var[Θ] = αβ/((α+β)²(α+β+1)) Now, begin looking for posterior: fΘ|X(θ|x) = (h(θ)(fX|Θ(x|θ)))/fX(x) Numerator = Γ(α+β)/(Γ(α)Γ(β))θᵃ⁻¹(1-θ)ᵇ⁻¹I (0 < θ < 1)*(n!/(x!(n-x)!)θˣ(1-θ)ⁿ⁻ˣ fΘ|X(θ|x) = Numerator/fX(x) where fX(x) does not depend on θ and therefore is not needed. keeping only terms relevant to θ: fΘ|X(θ|x) ∝ θ⁽ᵃ⁺ˣ⁾⁻¹(1-θ)⁽ᵇ⁺ⁿ⁻ˣ⁾⁻¹ Notice: θ⁽ᵃ⁺ˣ⁾⁻¹(1-θ)⁽ᵇ⁺ⁿ⁻ˣ⁾⁻¹ has same pattern as beta distributions. Posterior ~ Beta(α+x,β+n-x) where θ^b = (α+x)/(α+x+β+n-x) = (α+x)/(α+β+n)
Fisher Information
Part of the Cramer-Rao Lower Bound; ϝ(θ) = E[(d/dθ(ln(fX(X|θ)))²] from plain-text: the expectation of the squared value of the derivative of the natural log of the density
Implications of: Let X1, X2, . . . , Xn be the observations of a random sample from a distribution with mean value μ and standard deviation σ.
Sample Mean - 1. E[x̄] = E[x] = µ V[x̄] = (V(X1) +...+V(Xn))/n² = (nσ²)/n² = σ²/n Sample Variance - 2. =Σ(xi-x̄)²/(n-1) Sample Standard Deviation - 3. σsubscript(x̄) = σ/√(n)
HW3 Q3: If X₁,X₂,...Xₙ constitute a random sample of size n from a population given by: g(x;θ) = { (1/θ)exp(-(x-δ)/θ) for x > δ, { 0 elsewhere Find estimators for δ and θ by the method of moments. This distribution is sometimes referred to as the two-parameter exponential distribution.
Set new random variable Z = X - δ :: X = Z + δ E[X] = E[Z + δ] = E[Z] + E[δ] = E[Z] + δ where Z ~ (1/θ)exp(-z/θ) = θ + δ Write MME: E[X] = x̄ :: θ + δ = x̄ E[X²] = n⁻¹Σ(Xi)² By variance: E[X²] = Var[X] + (E[X])² = θ² + (θ + δ)² :: θ² + (θ + δ)² = n⁻¹Σ(Xi)² BUT: we know that θ + δ = x̄ so (θ + δ)² = x̄² :: θ² + x̄² = n⁻¹Σ(Xi)² θ² = n⁻¹Σ(Xi)² - x̄² θ^ₘₘₑ = sqrt(n⁻¹Σ(Xi)² - x̄²) then from E[X]: δ^ₘₘₑ = x̄ - sqrt(n⁻¹Σ(Xi)² - x̄²) [δ^ₘₘₑ, θ^ₘₘₑ] = [x̄-sqrt(n⁻¹Σ(Xi)²-x̄²), sqrt(n⁻¹Σ(Xi)² - x̄²)]
Prove x̄ is a biased estimator of δ, given: fX(x) = exp(-(x - δ)) i (x > δ), -∞ < δ < ∞
Setup: Let Z ~ Exponential(1). F(Z) = e⁻ᶻ I (z > 0) Then exp(-(x-δ)) = exp(-z) → X = Z + δ Step 1) cdf F(x) = P(X ≤ x) = P(Z + δ ≤ x) = P(Z ≤ x - δ) = FZ(x - δ) Step 2) pdf f(x) = dFX(x)/dx = fZ(x - δ) = e⁻⁽ˣ⁻δ⁾ I (x > δ) Step 3) from X = Z + δ and Z ~ exp(1), E[Exp(1)] = λ = 1: E[x̄] = E[Z^] + δ → 1 + δ x̄ is a biased estimator. Note: From this expression of E[x̄] = E[Z^] + δ → 1 + δ we can find that E[x̄ - λ] = E[x̄] - λ = E[Z^] + δ - λ = λ - λ + δ = δ = x̄ - λ is an unbiased estimator.
Prove that X/n is an unbiased estimator of θ, θ ∈ [0,1]; given X ~ Binomial(θ,n).
Setup: Binomial X is defined as X = ΣBt, where Bt are iid ~ Bernoulli(θ) RVs, θ ∈ [0,1]. and Σ is from t=1 to n So: E[Bt] = 1P(Bt = 1) + 0P(Bt = 0) = P(Bt = 1) = Probability of Success = θ V(Bt) = θ(1-θ) a.k.a. X ~ Binomial(θ,n) = "X ~ a set of n identically and independently distributed Bernoulli trials with probability of success θ." Step 1) Consider the density of a binomial distribution f(x) = f(x|θ,n) = nCx(p^x)(q^(n-x)) where: nCx = n!/(x!(n-x)!) p = θ q = (1-θ) Step 2) We know that the mean and variance of a binomial distribution is: E[X] = nθ and Var[X] = nθ(1-θ) i.e. X = ΣBt E[X] = ΣE[Bt] E[X] = n(θ) Var[X] = ΣV(Bt) where Σ is from t=1 to n Var[X] = n(θ(1-θ)) Step 3) Take Expectation wrt θ of X/n E[X/n] = n^-1E[X] = n^-1(nθ) = θ, θ ∈ [0,1]. ALWAYS REMEMBER SUPPORT! X/n is an unbiased estimator of θ. Also; X/n is the known sample mean estimate for n Bernoulli trials, even 1. i.e. X/n = X/1 = X is the unbiased estimator of θ in the case of a single Bernoulli(θ).
Prove that x̄ is an unbiased estimator in the case of X ~ Exponential(λ), λ > 0
Setup: remember x̄ is n⁻¹ΣXi where Σ is from i=1 to n and λ ∈ A, := [0,∞] Step 1) Consider the density of an exponential distribution → f(x) = fλ(x) = f(x|λ) = λexp(-λx) Step 2) We know that the mean and variance of an exponential distribution is E[x] = λ and Var[x] = λ² Step 3) Take Expectation wrt λ of x̄ → E[x̄] = E[n⁻¹ΣXi] → substitute x̄ = n⁻¹E[ΣXi] → Move n⁻¹ out of E = n⁻¹ΣE[Xi] → Move E inside Σ = n⁻¹Σ(λ) → Evaluate E[Xi] = n⁻¹(n * λ) → Evaluate Σ = λ, λ > 0 → Cancel n and n⁻¹ E[x̄] = E[x]; The estimator is unbiased. ALWAYS REMEMBER SUPPORT!
HW2 Q6: If θ₁^ and θ₂^ are unbiased estimators of parameter θ and Var[θ₁^] = 3Var[θ₂^], Find the constants a₁ and a₂ such that a₁θ₁^ + a₂θ₂^ is an unbiased estimator with minimum variance for such a linear combination.
Start with what we know: E[a₁θ₁^ + a₂θ₂^] = a₁E[θ₁^] + a₂E[θ₂^] = a₁(θ) + a₂(θ) = (a₁ + a₂)θ condition: a₁ + a₂ = 1. Now look at var: Var[a₁θ₁^ + a₂θ₂^] = a₁²Var[θ₁^] + a2²Var[θ₂^] Given: Var[θ₁^] = 3Var[θ₂^], = 3a₁²Var[θ₂^] + a₂²Var[θ₂^] = (3a₁² + a₂² )Var[θ₂^] Now, use condition from earlier that a₁ + a₂ = 1 → a₁ = 1 - a₂ (3(1-a₂)² + a₂²)Var[θ₂^] = 3(1 -2a₂ + a₂²) + a₂²)Var[θ₂^] = (3 - 6a₂ + 4a₂²)Var[θ₂^] Meaning: Variance is minimized based on the coefficient. da₂/dx (3 - 6a₂ + 4a₂²) = -6 + 8a₂ a₂ = 6/8 = 3/4 meaning → (a₁,a₂) = (1/4,3/4) for the minimum variance unbiased estimator a₁θ₁^ + a₂θ₂^ of θ when the conditions of a₁,a₂ are met.
Consider a Normal(µ,σ²) r.v. X with an unknown parameter of interest µ ∈ (-∞, ∞) Establish that x̄ is an efficient unbiased estimator of µ.
Step 1) E[x̄] = E[n⁻¹ΣXi] where Σ is from i=1 to n = n⁻¹E[ΣXi] = n⁻¹ΣE[Xi] = n⁻¹(nµ) = µ x̄ is an unbiased estimator Step 2) V[x̄] = V[n⁻¹ΣXi] where Σ is from i=1 to n = n⁻²V[ΣXi] = n⁻²ΣV(Xi) = n⁻²(nV(X)) = V(X)/n = σ²/n = Sample Variance (which makes sense, because x̄ is the sample mean) Step 3) -- work from inward to outward when looking at the expression, don't worry about how daunting it looks 3.1: Density fˣ(x|µ) = (1/sqrt(2*π*σ²))exp(-(x-µ)²/2σ²) I (-∞ < x < ∞) this should be memorized as the density of a normal distribution 3.2: Natural Log ln(fˣ(x|µ)) = -1/2ln(2*π*σ²) - 1/2(x-µ)²/σ² 3.3: Derivative w.r.t parameter ln(fˣ(x|µ)) → µ will be the parameter = (x - µ)/σ² 3.4 Squared Value = ((x - µ)/σ²)² 3.5 Expectation E[(x - µ)²/σ^4] = σ²/σ^4 = 1/σ² Step 4) 1/(n*ϝ(µ)) = 1/(n(1/σ²)) = σ²/n Step 5) Variance is equal to the CRLB. x̄ is an efficient estimator of µ.
Consider a sample of size n from Bernoulli(θ) RV X. θ contained within (0,1). Find the MLE.
Step 1) Write L(θ): L(θ) = Π(θˣʲ(1-θ)¹⁻ˣʲ I (Xj ∈ {0,1})) While you may continue to step 2, you can also use the log-likelihood for an easier attempt at solving. ln(L(θ)) = ln(θ^(ΣXj)(1-θ)^(n-ΣXj)) ΣXj(ln(θ)) + Σ(1 - Xj)(ln(1-θ)) l(θ) = ΣXj(ln(θ)) + (n - ΣXj)(ln(1-θ)) Notice that l(θ) denotes the log-likelihood compared to likelihood function L(θ). Now, take derivative wrt θ: l'(θ) = ΣXj/θ - (n - ΣXj)/(1-θ) Set equal to 0: ΣXj/θ - (n - ΣXj)/(1-θ) = 0 times by θ: ΣXj - (n - ΣXj)θ/(1-θ) times by (1-θ): ΣXj - ΣXjθ -nθ + ΣXjθ = 0 = ΣXj - nθ = 0 = ΣXj = nθ = ΣXj/n θ^ₘₗₑ = ΣXj/n which note = x̄
A sample of size n is from an Exponential(λ), λ > 0. Find MLE
Step 1) Write L(λ): L(λ) = Π(λexp(-λXj)I(Xj)>0) = λⁿexp(-λΣXj)I(min Xj > 0) l(λ) = nln(λ) - λΣXj l'(λ) = n/λ - ΣXj set to 0: = n/λ - ΣXj = 0 = n/λ = ΣXj = λ = ΣXj/n λ^ₘₗₑ = x̄
Implications of: X1, X2, ... Xn are independent Normal(µ, σ²) AND σ is unknown for population
T(n-1) = (x̄ - µ)/(√(S²/n)) = √n(x̄ - µ)/S ~T-distribution with (n-1) d.f. Notice the similarity between T and Normal distribution (i.e. Normal: √(n)((x̄ - µ)/σ))
True or False: If a function is sufficient, does it ALWAYS have a Maximum Likelihood Estimator?
TRUE
True or False? An unbiased estimator is always equal to its variance
TRUE
Which is better, an estimator with a small MSE or a large MSE?
The E in MSE stands for error; the smaller the MSE, the better the estimator. i.e. An estimator with a smaller MSE is relatively more efficient.
Central Limit Theorem (CLT)
The distribution of sample averages tends to be normal(µ, σ²) as n → ∞, regardless of the shape of the process distribution √(n)((x̄ - µ)/σ) converges in distribution to a standard normal variable. NOTE: 'Standard Normal' = Normal(µ=0, σ²=1) = Z-score Table
Law of Large Numbers (LLN)
The larger the number of individuals that are randomly drawn from a population, the more representative the resulting group will be of the entire population: A sample mean will converge to µ as n → ∞ P(x̄ - E[x] | > ε) → 0 as n → ∞
Definition 10.2
The statistic Θ^ is a consistent estimator of the parameter θ if and only if for each c > 0: lim(n→∞)P[|Θ^ - θ| < c] = 1 Note that consistency is an asymptotic property of an estimator.
Factorization Theorem
The sufficient statistic relationship is not easy to solve; but a criterion called Factorization Theorem simplifies the problem. Consider a sample X1, .. Xn from X with the density or pmf fX(x|θ). A statistic T = T(X1, ... Xn) is sufficient if and only if: fX₁,X₂,...Xₙ(x₁,x₂,...xₙ|θ) = [g(T,θ)][h(x₁,x₂,...xₙ)] a.k.a: If the joint density/pmf can be written as a product of a function g(T,θ) in T and θ, and a function h(x₁,x₂,...xₙ) of observations that does not depend on θ. If both of these functions can be found, the statistic is sufficient.
Sufficient Estimator (Sufficiency)
The theory behind a 'sufficient' statistic comes from data reduction. i.e. Given X ~ Normal(µ,σ²), and variance is known: Any inference about µ may be based on x̄ in place of the underlying sample X₁,X₂,...Xₙ
QUIZ 1: Q3 Let Θ^ be an unbiased estimator of θ, and T is a sufficient statistic for the parameter θ. Consider a new estimator Θ₁^ = E[Θ^|T]. Is it unbiased?
To begin, How would we check if its unbiased? E[Θ^] = θ; E[Θ₁^] = E[E[Θ^|T]] =E[E[θ^|T]] due to sufficiency of T = E[Θ^] = θ Θ₁^ is an unbiased estimator of θ
Methods of Moments
Used to estimate a parameter, i.e. a population mean, by a sample mean.
Efficient Estimator (Efficiency)
Uses the Cramer-Rao Inequality to determine an unbiased estimator with the minimal variance. Otherwise known as UMVUE; or Uniformly Minimum-Variance Unbiased Estimator.
HW2 Q9: If X1, X2, ... Xn constitute a random sample of size n from an exponential population: Show that x̄ is a consistent estimator of the parameter θ.
Using Chebyshev's Inequality: P(|x̄-θ| > ε) ≤ Var[x̄]/ε² Solve for Var[x̄] given Expon(θ): Var[n⁻¹ΣXi] = n⁻²Var[ΣXi] = n⁻²[nθ²] NOTE: if λ was given, it'd be 1/(nλ²) .. = P(|x̄-θ| > ε) ≤ θ²/nε² Where θ²/nε² → 0 as n→∞. x̄ is a consistent estimator of θ
HW2 Q11: If X1 and X2 are independent random variables having binomial distributions with the parameters θ and n1, and θ and n2: Show that (X1 + X2)/(n1 + n2) is a sufficient estimator of θ.
Using factorization theorem: fx1,x2(x1,x2) = fx1(x1)fx2(x2) = (n1!/x1!(n1-x1)!)θ^x1(1-θ)^(n1-x1) * (n2!/x2!(n2-x1)!)θ^x2(1-θ)^(n2-x2) = [n1!n2!/(x1!x2!(n1-x1)!(n2-x2)!) *θ^(x1+x2)*(1-θ)^(n1+n2-x1-x2)]*[1] =f[x]g[x] X1 + X2 is a sufficient estimator of θ, meaning that (X1+X2)/(n1+n2) is also a sufficient estimator of θ.
Consider a sample of size n from X ~ Exponential(2λ), λ > 0. Find MME of λ.
Using first moment: E[X] = x̄, E[X] = 2λ :: λ^ₘₘₑ = x̄/2
Show that sample mean x̄ is a consistent estimator of population mean µ, given: Population variance σ² is finite.
Using the Chebyshev Inequality: P(|x̄ - µ| > ε) ≤ E[(x̄ - µ)²]/ε² = Var[x̄]/ε² = (σ²/n)/ε² = σ²/(nε²) → 0 as n → ∞ x̄ is a consistent estimator of the population mean. Note how this proof mirrors LLN, but uses a stronger assumption: σ² < ∞ instead of E[|X|] < ∞
Let x̄₁ and x̄₂ be sample means based on samples of sizes n₁ and n₂ from Normal(µ,σ²). Show that the variance of estimator µ^ = wx̄₁ + (1-w)x̄₂ is a minimum when w = n₁/(n₁+n₂).
V(µ^) = V(wX₁^ + (1-w)X₂^) = V(wX₁^) + V((1-w)X₂^) = w²σ²/n₁ + (1-w)²σ²/n₂ We want the extremes, so take derivative w.r.t w and set to = 0: 2σ²(w/n₁ - (1-w)/n₂) = 0 w* = n₁/(n₁+n₂) then check with second derivative for max or min
Variance of an Unbiased Estimator θ^ of parameter θ Equation
V[θ^] = E[θ^ - E[θ^]]² = E[(θ^ - θ)²]
QUIZ 1: Q1 Consider a sample X₁, X₂, ... Xₙ from a normal population with mean µ and variance σ₁² (Read as: X ~ Normal(µ,σ₁²)) Find the variance of a random variable Y described as: Y = X₁ + 2¹/²X₂ + ... + n¹/²Xₙ
Var[Y] = Var[X₁ + 2¹/²X₂ + ... + n¹/²Xₙ] = Var[Σi¹/²Xᵢ],Σ ranges from i = 1:n = ΣVar[i¹/²Xᵢ] by law of variance: = ΣiVar[Xi] = Σiσ₁² = σ₁²Σi IDENTITY: Where it is known that Σx, for x = 1:n, = (n(n+1))/2 = σ₁²(n(n+1))/2
Variance of the Sample Mean
Var[x̄] =σ²/n Also: = E[(x̄-µ)²]
HW2 Q5: Show that the mean of a random sample of size n is a minimum variance unbiased estimator of the parameter λ in a Poisson population.
What this question MEANS: Show that x̄ is a UMVUE of Poisson(λ). E[x̄] = E[n^-1ΣXi] = n^-1(nλ) = λ Var[x̄] = Var[n^-1ΣXi] = n^-2Var[ΣXi] = n^-2ΣVar[Xi] = n^-2(nVar[Xi]) = Var[Xi]/n = λ/n now Fisher Information: fX(x|λ) = λ^x(exp(-λ))/x!; λ > 0 1) ln ln(fX(x|λ)) = xln(λ) - λ - ln(x!) 2) d/dx wrt λ x/λ - 1 3) ² ((x/λ - 1)² = (x/λ)² -2(x/λ) + 1 4) E[x] E[(x/λ)² -2(x/λ) + 1] = E[(x/λ)²] -2E[x/λ] + 1 = 1/λ²E[x²] -(2/λ)E[x] + 1 = 1/λ²E[x²] -2(λ/λ) + 1 = 1/λ²E[x²] - 1 Now: E[x²] = Var[x] + E[x]² = λ + λ² So: 1/λ²E[x²] - 1 = 1/λ²(λ + λ²) - 1 = 1/λ + 1 - 1 Finally; our fisher information is: = 1/λ using CRLB: 1/n(1/λ) = λ/n Var[x̄] ≥ λ/n Here, Var[x̄] = λ/n, meaning: x̄ is UMVUE of Poisson(λ)
HW3 Q1: Given a random sample of size n from an exponential population, use the method of moments to find an estimator of the parameter θ.
X ~ Exp(θ), pdf(compare to λ): (1/θ)exp(-x/θ) E[X] = θ Now MME: E[X] = x̄ E[X²] = n⁻¹Σ(Xi)² θ = x̄ θ^ₘₘₑ = x̄ Note that only E[X] is necessary.
HW3 Q5: Given a random sample of size n from a normal population with the known mean µ, find the maximum likelihood estimator for σ
X ~ Normal(µ,σ²) MLE: L(σ) = Π((2*pi*σ²)⁻¹/²exp(-1/2((x-µ)/σ)²)) (2*pi*σ²)⁻ⁿ/²exp(-(1/2)Σ((x-µ)/σ)²) ln(L(σ)) = - (n/2)ln(2*pi*σ²) - (1/2)Σ((x-µ)/σ)²) Take derivative w.r.t to σ², set to 0: -(n/2)/σ² + (1/2σ⁴)Σ(x-µ)² = 0 times both sides by σ²: -(n/2) + (1/2σ²)Σ(x-µ)² n/2 = (1/2σ²)Σ(x-µ)² n = Σ(x-µ)²/σ² σ² = n⁻¹Σ(x-µ)² σ²^ₘₗₑ = n⁻¹Σ(x-µ)²
HW3 Q2: Given a random sample of size n from a Poisson population, use the method of moments to obtain an estimator for the parameter λ
X ~ Poisson(λ) E[X] = λ MME: E[X] = x̄ E[X²] = n⁻¹Σ(Xi)² λ = x̄ λ^ₘₘₑ = x̄
Y has a Chi-Square distribution with ν degrees of freedom (d.f.) IF:
Y = ΣX² where X1, X2, .. Xv are independent standard Normal. where Σ is from i=1 to v NOTE: 'Standard Normal' = Normal(µ=0, σ²=1) = Z-score Table
HW2 Q7: If x̄₁ is the mean of a random sample of size n from a normal population with the mean µ and the variance σ₁², x̄₂ is the mean of a random sample size n with mean µ and variance σ₂², and the two samples are independent, show that: a) wx̄₁ +(1-w)x̄₂, where 0 ≤ w ≤ 1, is an unbiased estimator of µ. :: NOTE: x̄₁,x̄₂ have same mean, but different variance. b) the variance of this estimator is a minimum when: w = σ₂²/(σ₁² + σ₂²)
a) E[wx̄₁ +(1-w)x̄₂] = E[wx̄₁] + E[(1-w)x̄₂] = wE[x̄₁] + (1-w)E[x̄₂] = w(µ) + (1-w)(µ) = (w + 1 - w)(µ) = µ wx̄₁ +(1-w)x̄₂, where 0 ≤ w ≤ 1; is an unbiased estimator of µ. b) Var[wx̄₁ +(1-w)x̄₂] = Var[wx̄₁] + Var[(1-w)x̄₂] = w²Var[x̄₁] + (1-w)²Var[x̄₂] = w²(σ₁²/n) + (1-w)²(σ₂²/n) = n⁻¹(w²σ₁² + (1-w)²σ₂²) = n⁻¹(w²σ₁² + (1-2w+w²)σ₂² = n⁻¹(w²σ₁² + σ₂² - 2wσ₂² + w²σ₂²) = n⁻¹(w²(σ₁²+σ₂²) - 2wσ₂² + σ₂²) d/dx with respect to w (minimum variance always requires derivative!!!): = n⁻¹(2w(σ₁²+σ₂²) - 2σ₂²) now, set = to 0. → n⁻¹(2w(σ₁²+σ₂²) - 2σ₂²) = 0 → (2w(σ₁²+σ2²) - 2σ₂²) = 0*n = 0 → 2w(σ₁²+σ₂²) = 2σ₂² → w(σ₁²+σ₂²) = σ₂² w = σ₂²/(σ₁²+σ₂²)
Bias
b(θ) = E[θ^] - θ is called the bias of the estimator θ^. Bias is 0 for all θ ∈ A of an unbiased estimator.
Asymptotically Unbiased Estimator
if max_θ ∈ A_ b(θ) → 0 as n → ∞ a.k.a if the bias of the estimator approaches 0 as n increases
HW2 Q4: Show that, if θ^ is an unbiased estimator of θ and var[θ^] != 0: (θ^)² is not an unbiased estimator of θ².
if θ^ is an unbiased estimator, E[θ^] = θ. Var[θ^] = E[(θ^)²] + E[θ^]² → E[(θ^)²] = Var[(θ^)] + E[θ^]² = Var[(θ^)] + θ² It is given that var[θ^] != 0, meaning that θ^² is a BIASED estimator for θ².
Suppose that you observe a sample of 49 Poisson RVs with mean λ = 10. Formulate the LLN and the CLT
n = 49 iid Pois(10) r.vs X1, ... X49 µ = E{X} = 10, σ² = V(X) = 10, σ = √(10) x̄ = n^-1ΣXi = 1/49 (X1 + X2 + ... +X49) where Σ is from i=1 to n LLN: P(|x̄ − 10| > ε) → 0 as n → ∞. CLT: √(n)((x̄ - µ)/σ) = 7(x̄ − 10)/√10 i.e. the cdf converges to the cdf of a standard normal variable. NOTE: 'Standard Normal' = Normal(µ=0, σ²=1) = Z-score Table
Sample Standard Deviation
s = sqrt(s²)
Sample Mean
x̄ = (X1 + X2 +...+Xn)/n = n⁻¹ΣXi where Σ is from i=1 to n
Z-Scoring !!with a sample!!
~N(0,1) i.e. µ = 0, σ² = 1 -> σ = 1 Z = ((x̄ - µ)/σx) = ((x̄ - µ)/(σ/√(n))) = √(n)((x̄ - µ)/σ) notice that σx is the sample standard deviation (i.e. √(σ²/n) and x̄ is the sample mean. ('SAMPLE'!!) If problem doesn't involve a sample, it'd be (x-µ)/σ and wouldn't need to include √(n).
HW2 Q12: If X1, X2, ... Xn constitute a random sample of size n from a geometric population: Show that Y = X1 + X2 + .. + Xn is a sufficient estimator of the parameter θ. NOTE THAT: X1 + X2 + .. + Xn = Σxi
Π(((1-θ)^x)(θ)) = θ^n(1-θ)^(Σxi) = [θ^n(1-θ)^(Σxi)]*[1] = f[x]g[x] Σxi is a sufficient estimator of the parameter θ.
HW2 Q10: If X1, X2, ... Xn constitute a random sample of size n from an exponential population: Show that x̄ is a sufficient estimator of the parameter θ.
Π((1/θ)exp(-xi/θ)) where Π is from i=1 to n = ((1/θ)^n)exp(-Σ(xi/θ))Π = ((1/θ)^n)exp(-nx/θ)Π = [((1/θ)^n)exp(-nx/θ)][Π] = f[x]g[x] x̄ is a sufficient estimator of θ by factorization theorem.
QUIZ 1: Q5 Consider a sample of size n from Gamma(α, β). Let α be given. Find the (minimal) sufficient statistic for parameter β.
Π((βᵃ/Γ(α))xᵃ⁻¹exp(-βxi) = (βᵃ/Γ(α))ⁿxⁿᵃ⁻ⁿexp(-βΣxi) meaning T = Σxi is a sufficient statistic, meaning x̄ is also a sufficient statistic.
HW3 Q7: Let X₁,X₂,...Xₙ be a random sample of size n from a uniform population given by f(x;θ)= {1 for θ - 1/2 < x < θ + 1/2 {0 elsewhere Check whether the following estimators aare maximum likelihood estimators of θ. A) 1/2(X₍₁₎ + X₍ₙ₎) B) 1/3(X₍₁₎ + 2X₍₂₎)
θ - 1/2 < X < θ + 1/2 + 1/2 to each: θ < X + 1/2 < θ + 1 where θ < X + 1/2 is noted. -1/2 to each: θ - 1 < X - 1/2 < θ where X - 1/2 < θ is noted. combine to create bounds for θ: X - 1/2 < θ < X + 1/2 Now consider 1 through n: X₍₁₎ - 1/2 ≤ X₍ₙ₎ - 1/2 < θ θ < X₍₁₎ + 1/2 ≤ X₍ₙ₎ + 1/2 :: X₍ₙ₎ - 1/2 < θ < X₍₁₎ + 1/2; where θ = θ^ estimator, meaning we want to prove for each estimator θ^: X₍ₙ₎ - 1/2 < θ^ θ^ < X₍₁₎ + 1/2; Now consider given estimators. A) Try to create A with given equalities: X₍ₙ₎ - 1/2 < θ^ < X₍₁₎ + 1/2 + X₍ₙ₎ - 1/2 to each side: 2X₍ₙ₎ - 1 < θ+ X₍ₙ₎ - 1/2 < X₍₁₎ + X₍ₙ₎ / by 2: X₍ₙ₎ - 1/2 < ... < (X₍₁₎ + X₍ₙ₎)/2 -> right side of inequality proved + X₍₁₎ + 1/2 to each side: X₍₁₎ + X₍ₙ₎ < θ + X₍₁₎ + 1/2 < 2X₍₁₎+1 / by 2: (X₍₁₎ + X₍ₙ₎)/2 < ... < X₍₁₎ + 1/2 -> left side of inequality proved. A) is valid MLE of parameter θ. B) Try to create B with given inequalities: X₍ₙ₎ - 1/2 < θ^ < X₍₁₎ + 1/2 by definition of estimator: θ^ - X₍ₙ₎ + 1/2 < X₍₁₎ - X₍ₙ₎ + 1 +2X₍₂₎-1/2: =X₍ₙ₎+2X₍₂₎-1 <θ^+2X₍₂₎-1/2 <X₍₁₎+2X₍₂₎ (X₍ₙ₎+2X₍₂₎-1)/3 <(θ^+2X₍₂₎- 1/2)/3 < (X₍₁₎+2X₍₂₎)/3 +1/2-X₍ₙ₎: X₍₁₎+2X₍₂₎ < θ^ -X₍ₙ₎ +X₍₁₎+2X₍₂₎ + 1/2 < X₍₁₎ - X₍ₙ₎ + 1 + X₍₁₎+2X₍₂₎ (θ^+2X₍₂₎- 1/2)/3 < (X₍₁₎+2X₍₂₎)/3 < (θ^ -X₍ₙ₎ +X₍₁₎+2X₍₂₎ + 1/2)/3 -2X₍₂₎ - 1/3: θ^/3 - 1/3 < X₍₁₎/3 - 1/3 < (θ^ -X₍ₙ₎ +X₍₁₎ )/3 *3: θ^- 1 < X₍₁₎ < +X₍₁₎ + θ^ -X₍ₙ₎ Look at right half; X₍₁₎ < +X₍₁₎ + θ^ -X₍ₙ₎ X₍ₙ₎ < θ^ But we know from original inequality that the bounds are: X₍ₙ₎ - 1/2 < θ^ Meaning that X₍ₙ₎ < θ^ is not always guaranteed to hold which is derived from the estimator; B) is invalid MLE of parameter θ.
Is λ^ = 5 an unbiased estimator for the case when X ~ Exponential(λ)? (As derived in #18, an unbiased estimator for Exponential(λ) is λ, λ > 0)
λ^ = 5 is BIASED for any X ~ Exponential(λ) where λ != 5. Exponential(λ) can range from 0 to ∞. If Exponential(λ) was specified to range from 5 to 5, λ^ is unbiased. Note how λ, λ > 0 calls for all numbers; its support is necessary to make it unbiased.
