Parametric model
","answer":"A family of distributions {f(x;θ):θ∈Θ} indexed by a parameter θ in a set Θ⊆Rk. The data X1,…,Xn are usually iid from one unknown member.
Statistic
","answer":"Any function T(X1,…,Xn) of the data that does not involve the unknown parameter. An estimator is a statistic used to guess θ.
Sample mean and sample variance
","answer":"Moments of the sample mean and variance
","answer":"Any iid sample with mean μ and variance σ2<∞; normality is not needed.
Chi-square law
","answer":"If Z1,…,Zν are iid N(0,1) then Z12+⋯+Zν2∼χν2. Independent chi-squares add: χa2+χb2∼χa+b2.
Student t law
","answer":"If Z∼N(0,1) and V∼χν2 are independent, then T=V/νZ∼tν.
F law
","answer":"If U∼χd12 and V∼χd22 are independent, then V/d2U/d1∼Fd1,d2. Also 1/Fd1,d2∼Fd2,d1 and tν2∼F1,ν.
Student's theorem (normal samples)
","answer":"For X1,…,Xn iid N(μ,σ2). Independence of X and S2 characterizes the normal family.
t statistic
","answer":"Follows from Student's theorem and the definition of t; σ cancels.
Variance ratio (two normal samples)
","answer":"Independent samples of sizes m and n from normal populations.
Chi-square
Degrees of freedom ν∈{1,2,…}, written χν2. Same law as Gamma with shape ν/2 and rate 1/2.
Student t
Degrees of freedom ν>0, written tν. Symmetric about 0 with heavier tails than N(0,1); tν→N(0,1) in distribution as ν→∞. ν=1 is the Cauchy law (no mean).
Does not exist: E[etX]=∞ for every t=0, and only moments of order below ν are finite.
F
Degrees of freedom d1,d2>0, written Fd1,d2. Here B(⋅,⋅) is the Beta function.
Does not exist: E[etX]=∞ for every t>0 because the right tail is polynomial.
Order statistic distributions
","answer":"X1,…,Xn iid with cdf F, sorted as X(1)≤⋯≤X(n), and k∈{1,…,n}. The density formula needs a pdf f; the max and min cdfs hold for any iid sample. Example: for Unif(0,θ), FX(n)(x)=(x/θ)n for 0≤x≤θ.
Sufficient statistic
","answer":"T(X) is sufficient for θ if the conditional distribution of X given T(X) does not depend on θ: once T is known, the rest of the data carries no information about θ.
Factorization theorem
","answer":"T is sufficient if and only if the joint pmf or pdf factors like this, with h free of θ. Watch indicator functions of the support, which may involve θ.
Minimal sufficient statistic
","answer":"A sufficient statistic that is a function of every other sufficient statistic, so it gives the greatest data reduction without losing information.
Minimal sufficiency criterion
","answer":"If this equivalence holds for all sample points x,y, then T is minimal sufficient.
Complete statistic
","answer":"T is complete if Eθ[g(T)]=0 for all θ forces Pθ(g(T)=0)=1 for all θ. No nontrivial function of T has mean zero everywhere.
Ancillary statistic
","answer":"V(X) is ancillary if its distribution does not depend on θ. Example: in a location family, the range X(n)−X(1).
Exponential family form
","answer":"h≥0 and the tj do not involve θ, and c(θ)>0. The support {x:f(x;θ)>0} cannot depend on θ, so Uniform(0,θ) is not an exponential family.
Natural sufficient statistic
","answer":"Always sufficient for an iid sample. It is also complete (hence minimal) when the set {(w1(θ),…,wk(θ)):θ∈Θ} contains an open set in Rk.
Rao–Blackwell theorem
","answer":"W is unbiased for τ(θ) and T is sufficient, so ϕ(T) is a genuine statistic. Conditioning on a sufficient statistic never hurts.
Lehmann–Scheffé theorem
","answer":"If T is a complete sufficient statistic and ϕ(T) is unbiased for τ(θ), then ϕ(T) is the unique UMVUE of τ(θ). Recipe: find any unbiased W and compute E[W∣T], or guess a function of T that is unbiased.
Basu's theorem
","answer":"If T is complete and sufficient and V is ancillary, then T and V are independent for every θ. Example: for N(μ,σ2) with σ2 known, X is independent of S2.
Bias
","answer":"θ^ is unbiased if the bias is 0 for every θ.
Mean squared error
","answer":"A biased estimator can have smaller MSE than an unbiased one.
Consistency
","answer":"A sequence of estimators θ^n is consistent if θ^npθ for every θ∈Θ.
Sufficient condition for consistency
","answer":"Equivalently, bias →0 and variance →0. Proof: Markov's inequality applied to (θ^n−θ)2.
UMVUE
","answer":"An unbiased estimator of τ(θ) whose variance is at most that of every other unbiased estimator of τ(θ), for every value of θ simultaneously.
Moment equations
","answer":"Set the first k sample moments equal to the population moments and solve for the k parameters.
Example: normal model
","answer":"Method-of-moments estimators are usually consistent but need not be efficient or functions of a sufficient statistic.
Maximum likelihood estimator
","answer":"θ^ maximizes the likelihood L(θ)=∏i=1nf(Xi;θ) over Θ, equivalently the log-likelihood ℓ(θ)=logL(θ).
Likelihood equation
","answer":"Check that the solution is a maximum and compare with the boundary. If the support depends on θ (for example Uniform(0,θ)), maximize directly instead of differentiating.
Invariance of the MLE
","answer":"The MLE of any function of θ is that function of the MLE.
Standard MLEs (one parameter)
","answer":"Normal MLEs
","answer":"σ^2 is biased but consistent; S2 is the unbiased version.
Likelihood and log-likelihood
","answer":"Viewed as a function of θ with the data fixed. Constant factors not involving θ can be dropped.
Score function
","answer":"Score has mean zero
","answer":"Under regularity: the support does not depend on θ and differentiation can pass under the integral.
Fisher information (one observation)
","answer":"The second form needs logf twice differentiable and the same regularity as for the score.
Information in an iid sample
","answer":"Reparametrization
","answer":"Information matrix
","answer":"Multiparameter version; it also equals the covariance matrix of the score vector.
Common information values
","answer":"Each is I1 for a single observation.
Cramér–Rao lower bound
","answer":"For any unbiased estimator T of τ(θ) under regularity. For τ(θ)=θ the bound is 1/(nI1(θ)). Equality holds only if U(θ)=a(θ)(T−τ(θ)).
Efficiency
","answer":"T is efficient if it attains the bound (efficiency 1).
Asymptotic normality of the MLE
","answer":"Under regularity conditions. So the MLE is consistent and asymptotically efficient, and τ(θ^n) is asymptotically normal by the delta method.
Delta method
","answer":"Given n(Tn−θ)dN(0,σ2) and τ differentiable at θ with τ′(θ)=0. For the MLE, under the regularity conditions above, σ2=1/I1(θ), so n(τ(θ^n)−τ(θ))dN(0,[τ′(θ)]2/I1(θ)).
Standard error of the MLE
","answer":"Expected information evaluated at θ^, or the observed information.
Confidence interval
","answer":"A random interval [L(X),U(X)] with Pθ(L≤θ≤U)≥1−α for every θ. The probability describes the procedure; a computed interval either contains θ or it does not.
Pivot
","answer":"A function Q(X,θ) whose distribution does not depend on θ. Find a,b with P(a≤Q≤b)=1−α and solve the inequalities for θ.
Quantile notation
","answer":"Upper-tail points: P(Z>za)=a for Z∼N(0,1), and similarly tν,a, χν,a2, Fd1,d2,a. So z0.025≈1.96.
z interval for a mean (σ known)
","answer":"Pivot σ/nX−μ∼N(0,1).
t interval for a mean (σ unknown)
","answer":"Pivot S/nX−μ∼tn−1.
Chi-square interval for a variance
","answer":"Pivot (n−1)S2/σ2∼χn−12; the larger quantile gives the lower endpoint. Sensitive to non-normality.
Pooled two-sample t interval
","answer":"Independent normal samples with a common variance.
Welch interval (unequal variances)
","answer":"Approximate; the degrees of freedom ν^ need not be an integer.
Paired t interval
","answer":"Interval for μD=μX−μY from n pairs (Xi,Yi). Different pairs are independent, but Xi and Yi in the same pair may be dependent; assume D1,…,Dn iid N(μD,σD2). D and SD are the sample mean and standard deviation of the Di: this is the one-sample t interval applied to them.
Interval for a variance ratio
","answer":"Large-sample Wald interval
","answer":"Based on asymptotic normality of θ^, often with se=1/In(θ^). Coverage can be poor for small n or p near 0 or 1.
Duality of intervals and tests
","answer":"This set is a 1−α confidence set. Conversely, reject H0:θ=θ0 exactly when θ0∈/C(x).
Decision rule
","answer":"A function δ mapping the data x to an action a, such as an estimate of θ. A loss function L(θ,a)≥0 measures the cost of action a when the truth is θ.
Common loss functions
","answer":"Risk function
","answer":"Average loss over repeated data at a fixed θ. Under squared loss the risk is the MSE.
Admissibility
","answer":"δ′ dominates δ if R(θ,δ′)≤R(θ,δ) for all θ, with strict inequality for some θ. A rule is admissible if no rule dominates it.
Minimax rule
","answer":"δ∗ minimizes the worst-case risk.
Bayes risk
","answer":"Risk averaged over a prior π (a sum for a discrete prior).
Bayes rule
","answer":"A rule minimizing r(π,δ). It can be found pointwise: for each observed x, choose the action minimizing the posterior expected loss E[L(θ,a)∣x].
Bayes rule under squared loss
","answer":"The posterior mean.
Beta–Binomial Bayes rule
","answer":"Prior p∼Beta(a,b) with a,b>0, and X∣p∼Bin(n,p). δπ is the posterior mean, the Bayes rule under squared loss. With a=b=n/2 its risk is constant, R(p,δπ)=4(n+1)21 for all p∈[0,1], so it is minimax.
Bayes rule under absolute loss
","answer":"Any median of the posterior.
Bayes rule under 0-1 loss
","answer":"The posterior mode, for a discrete parameter space.
Links between the criteria
","answer":"A unique Bayes rule is admissible. A Bayes rule whose risk is constant in θ is minimax. An admissible rule with constant risk is minimax.