Hypotheses and tests
","answer":"Test H0:θ∈Θ0 against H1:θ∈Θ1 (disjoint). A test is a rejection region R, or a test function ϕ(x)∈[0,1] giving the probability of rejecting.
Type I and Type II errors
","answer":"Type I: rejecting H0 when it is true. Type II: failing to reject H0 when H1 is true.
Power function
","answer":"For θ∈Θ0 it is the Type I error rate; for θ∈Θ1 it is one minus the Type II error rate.
Size and level
","answer":"p-value
","answer":"For a test that rejects for large T. Reject at level α exactly when p≤α. For a simple null and continuous T, p(X)∼Unif(0,1) under H0. It is not the probability that H0 is true.
Uniformly most powerful test
","answer":"A level-α test whose power is at least that of every other level-α test at every θ∈Θ1.
Neyman–Pearson lemma
","answer":"For simple H0:θ=θ0 versus simple H1:θ=θ1, with k≥0 and γ chosen so that Eθ0[ϕ]=α, this likelihood-ratio test is most powerful at level α.
Monotone likelihood ratio
","answer":"The family has MLR in T(x) if for every θ1<θ2 the ratio f(x;θ2)/f(x;θ1) is a nondecreasing function of T(x). A one-parameter exponential family with w(θ) increasing has MLR in ∑it(xi).
UMP one-sided test (Karlin–Rubin)
","answer":"With MLR in T, this is UMP of level α for H0:θ≤θ0 versus H1:θ>θ0. For discrete T, randomize at T=c to get size exactly α. Two-sided alternatives usually have no UMP test.
Likelihood ratio test
","answer":"0≤Λ≤1. The denominator uses the unrestricted MLE and the numerator the MLE under H0.
Duality of tests and confidence sets
","answer":"If, for each θ0, the acceptance region A(θ0) gives a level-α test of H0:θ=θ0, then Pθ(θ∈C(X))≥1−α for every θ: C is a 1−α confidence set. Conversely, a 1−α confidence set C gives the level-α acceptance regions A(θ0)={x:θ0∈C(x)}.
Consistency
","answer":"θ^npθ for every θ. Consistent estimators can still be badly biased for small n.
Asymptotic normality
","answer":"θ^n is asymptotically normal with asymptotic variance v(θ) if n(θ^n−θ)dN(0,v(θ)). Then θ^n is approximately N(θ,v(θ)/n).
Score and Fisher information
","answer":"For iid X1,…,Xn with density f(x;θ): the score is U(θ)=∂θ∂logL(θ;X)=∑i=1n∂θ∂logf(Xi;θ), and I1(θ)=Eθ[(∂θ∂logf(X;θ))2], In(θ)=nI1(θ). Under regularity (support free of θ, differentiation under the integral), Eθ[U(θ)]=0, Varθ(U(θ))=In(θ), and I1(θ)=−Eθ[∂θ2∂2logf(X;θ)].
Asymptotics of the MLE (vector parameter)
","answer":"Under regularity; I1 is the per-observation information matrix.
Delta method (first order)
","answer":"Given n(Tn−θ)dN(0,σ2) and g differentiable at θ with g′(θ)=0.
Delta method (second order)
","answer":"Same setup, when g′(θ)=0 and g′′(θ)=0 exists.
Multivariate delta method
","answer":"Given n(Tn−θ)dNk(0,Σ) and g:Rk→R differentiable at θ with ∇g(θ)=0.
Variance-stabilizing transformation
","answer":"When the asymptotic variance σ2(θ) depends on θ. Poisson example: n(X−λ)dN(0,1/4).
Wilks' theorem
","answer":"r=dimΘ−dimΘ0, the number of independent restrictions. Needs regularity and the true θ in the interior. Reject when −2logΛn>χr,α2.
Wald test
","answer":"Under H0:θ=θ0 (regularity, θ0 interior). Equivalently Z=(θ^−θ0)/se(θ^)dN(0,1). Vector form: (θ^−θ0)⊤In(θ^)(θ^−θ0)dχk2.
Wald confidence interval
","answer":"θ^ is the MLE. Coverage is approximately 1−α for large n, under the regularity conditions for asymptotic normality of the MLE (with Slutsky's theorem to plug in θ^). It is the set of θ0 that the level-α Wald test does not reject. The observed information may replace In(θ^). Not invariant to reparametrization.
Score (Rao) test
","answer":"Under H0:θ=θ0 (regularity). Here U(θ)=∂θlogL(θ;X) and In=nI1. Uses only quantities at θ0, so no unrestricted fit is needed.
Comparing the three tests
","answer":"Under H0 the likelihood ratio, Wald and score statistics are asymptotically equivalent, all with a χ2 limit. They can differ noticeably in small samples, and only the LR and score tests are invariant to reparametrization.
Bayes' theorem for densities
","answer":"The denominator is the marginal (prior predictive) density of the data.
Posterior is likelihood times prior
","answer":"Drop every factor not involving θ, then recognize the kernel of a known density.
Posterior predictive
","answer":"Distribution of a new observation x~, conditionally independent of the data given θ.
Credible interval
","answer":"A set C with P(θ∈C∣x)=1−α. Equal-tailed: between the posterior α/2 and 1−α/2 quantiles. Highest posterior density (HPD): the shortest such set for a unimodal posterior. The probability is about θ given the observed data.
Conjugate prior
","answer":"A family of priors is conjugate for a likelihood if the posterior stays in the same family; updating then only changes the parameters.
Bayes estimators
","answer":"Minimizes posterior expected loss. Squared-error loss: the posterior mean E[θ∣x], when E[θ2∣x]<∞. Absolute-error loss: any posterior median, when E[∣θ∣∣x]<∞. 0-1 loss: the posterior mode (MAP) for a discrete parameter; for a continuous parameter the mode is only a limit of Bayes rules for the losses 1{∣a−θ∣>ε} as ε→0.
Beta–Binomial
","answer":"Posterior mean α+β+nα+x, a weighted average of the prior mean and x/n. Predictive probability that one new trial succeeds: the same value.
Gamma–Poisson
","answer":"Gamma with shape and RATE. Posterior mean (α+∑xi)/(β+n).
Normal–Normal (known variance)
","answer":"τn21=τ021+σ2n and μn=τn2(τ02μ0+σ2nx): precisions add, and the mean is a precision-weighted average.
Gamma–Exponential
","answer":"Exponential with RATE λ; Gamma with shape and RATE.
Dirichlet–Multinomial
","answer":"Posterior mean of pj is (αj+xj)/(∑iαi+n). With k=2 this is Beta–Binomial.
Jeffreys prior
","answer":"Invariant under reparametrization: the Jeffreys prior for η=g(θ) is the transformed Jeffreys prior for θ.
Jeffreys prior examples
","answer":"The Bernoulli case is Beta(21,21). For the normal sd, the mean is taken as known.
Improper prior
","answer":"A prior with ∫π(θ)dθ=∞. It can still be used if the resulting posterior is a proper density; always check this.
Linear model
","answer":"X is a known n×p design matrix of full column rank p (including the intercept column if there is one).
Normal equations
","answer":"Least squares estimator
","answer":"Minimizes ∥y−Xβ∥2. With normal errors it is also the MLE.
Hat matrix, fitted values, residuals
","answer":"Properties of the hat matrix
","answer":"H is the orthogonal projection onto the column space of X; I−H projects onto its orthogonal complement and has trace n−p.
Mean and variance of the LSE
","answer":"Unbiased estimator of the error variance
","answer":"Divide by n−p, not n: fitting p coefficients uses up p degrees of freedom.
Normal errors: exact distributions
","answer":"When ε∼Nn(0,σ2In); also β^ and s2 are independent.
Confidence interval for a mean response
","answer":"Interval for E[y0]=x0⊤β at a fixed new row x0∈Rp, assuming ε∼Nn(0,σ2In). Uses Var(x0⊤β^)=σ2x0⊤(X⊤X)−1x0 and the independence of β^ and s2.
Prediction interval for a new response
","answer":"Covers y0=x0⊤β+ε0 with probability 1−α, where ε0∼N(0,σ2) is independent of ε∼Nn(0,σ2In). The extra 1 is the variance of ε0, so the interval is wider than the one for E[y0] and does not shrink to zero width as n→∞.
Gauss–Markov theorem
","answer":"Under E[ε]=0 and Var(ε)=σ2In (normality not needed), for every vector c the estimator c⊤β^ has the smallest variance among all linear unbiased estimators of c⊤β: least squares is BLUE.
Sum of squares decomposition
","answer":"Holds when the model contains an intercept.
Coefficient of determination
","answer":"Fraction of variation explained (model with intercept). R2 never decreases when predictors are added; the adjusted version can.
F test for nested models
","answer":"Full model with pF coefficients, reduced model (the null) with pR<pF nested inside it, normal errors. Reject for large F.
Empirical CDF
","answer":"For each fixed x, nF^n(x)∼Bin(n,F(x)), so E[F^n(x)]=F(x) and Var(F^n(x))=F(x)(1−F(x))/n.
Glivenko–Cantelli theorem
","answer":"Uniform convergence, for any distribution F and iid data.
DKW inequality
","answer":"Gives a distribution-free confidence band: F^n(x)±log(2/α)/(2n).
Kolmogorov–Smirnov statistic
","answer":"Tests H0:F=F0. For continuous F0 its null distribution does not depend on F0, and nDn converges in distribution to the Kolmogorov distribution. Two-sample version: supx∣F^m(x)−G^n(x)∣.
Sign test
","answer":"Needs only continuity. E0[S]=n/2 and Var0(S)=n/4. Discard observations equal to m0.
Wilcoxon signed-rank statistic
","answer":"Ri is the rank of ∣Di∣ among ∣D1∣,…,∣Dn∣. Assumes the distribution is continuous and symmetric about m0 under H0. For paired data use the within-pair differences.
Signed-rank null mean and variance
","answer":"Without ties. For large n compare (W+−E0W+)/Var0W+ with N(0,1).
Wilcoxon rank-sum statistic
","answer":"Ri is the rank of Xi in the combined sample X1,…,Xm,Y1,…,Yn. Null hypothesis: both samples come from the same continuous distribution.
Rank-sum null mean and variance
","answer":"Without ties; with ties use midranks and a corrected variance.
Mann–Whitney form
","answer":"Same variance as W. U/(mn) estimates P(X>Y).
Bootstrap principle
","answer":"Replace the unknown F by the empirical distribution F^n. The sampling behaviour of θ^−θ under F is approximated by that of θ^∗−θ^ under F^n, where θ^∗ is computed from n draws with replacement from the data.
Bootstrap standard error
","answer":"θ^1∗,…,θ^B∗ are the estimates from B bootstrap resamples.
Bootstrap bias estimate
","answer":"Percentile interval
","answer":"Lower and upper empirical quantiles of the B bootstrap replicates.
Basic (pivotal) interval
","answer":"Uses the bootstrap distribution of θ^∗−θ^ as a stand-in for that of θ^−θ. The normal interval is θ^±zα/2seboot.
Permutation test
","answer":"When H0 makes the observations exchangeable (for example, two samples from the same distribution), the group labels carry no information. Recompute the statistic T under relabellings and see where the observed value Tobs falls.
Exact permutation p-value
","answer":"Π is the set of all relabellings (including the observed one), for a test rejecting for large T.
Monte Carlo permutation p-value
","answer":"B random relabellings. Counting the observed labelling (the +1 terms) keeps the test valid at its nominal level.
Monte Carlo estimate and its standard error
","answer":"Yj iid simulated draws and sg the sample standard deviation of the g(Yj). Error shrinks like 1/M.
Standard error of a simulated proportion
","answer":"Applies to a Monte Carlo p-value or coverage estimate from B replications.
Two sources of error
","answer":"A bootstrap answer has statistical error from the finite sample size n and Monte Carlo error from the finite number of resamples B. Increasing B removes only the second.