Mathematical Statistics I

Formula sheet

Definitions, key results and distributions, unit by unit.

Practice Mathematical Statistics I problems Drill these formulas

Jump to a unit

Unit 1: Parametric models and sampling distributions
Unit 2: Sufficiency, completeness, and exponential families
Unit 3: Bias, risk, consistency, and point estimation
Unit 4: Likelihood, score, and Fisher information
Unit 5: Pivots and confidence sets
Unit 6: Loss, risk, and introductory decision theory

Unit 1

Parametric models and sampling distributions

Models and statistics

Parametric model

A family of distributions $\{f(x;\theta) : \theta \in \Theta\}$ indexed by a parameter $\theta$ in a set $\Theta \subseteq \mathbb{R}^k$ . The data $X_1,\dots,X_n$ are usually iid from one unknown member.

Statistic

Any function $T(X_1,\dots,X_n)$ of the data that does not involve the unknown parameter. An estimator is a statistic used to guess $\theta$ .

Sample mean and sample variance

\overline{X} = \frac{1}{n}\sum_{i=1}^n X_i, \qquad S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \overline{X})^2

Moments of the sample mean and variance

E[\overline{X}] = \mu, \qquad \operatorname{Var}(\overline{X}) = \frac{\sigma^2}{n}, \qquad E[S^2] = \sigma^2

Any iid sample with mean $\mu$ and variance $\sigma^2 < \infty$ ; normality is not needed.

Normal sampling theory

Chi-square law

If $Z_1,\dots,Z_\nu$ are iid $N(0,1)$ then $Z_1^2 + \cdots + Z_\nu^2 \sim \chi^2_\nu$ . Independent chi-squares add: $\chi^2_{a} + \chi^2_{b} \sim \chi^2_{a+b}$ .

Student t law

If $Z \sim N(0,1)$ and $V \sim \chi^2_\nu$ are independent, then $T = \frac{Z}{\sqrt{V/\nu}} \sim t_\nu$ .

F law

If $U \sim \chi^2_{d_1}$ and $V \sim \chi^2_{d_2}$ are independent, then $\frac{U/d_1}{V/d_2} \sim F_{d_1,d_2}$ . Also $1/F_{d_1,d_2} \sim F_{d_2,d_1}$ and $t_\nu^2 \sim F_{1,\nu}$ .

Student's theorem (normal samples)

\overline{X} \sim N\!\left(\mu, \frac{\sigma^2}{n}\right), \qquad \frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}, \qquad \overline{X} \perp S^2

For $X_1,\dots,X_n$ iid $N(\mu,\sigma^2)$ . Independence of $\overline{X}$ and $S^2$ characterizes the normal family.

t statistic

\frac{\overline{X} - \mu}{S/\sqrt{n}} \sim t_{n-1}

Follows from Student's theorem and the definition of $t$ ; $\sigma$ cancels.

Variance ratio (two normal samples)

\frac{S_X^2/\sigma_X^2}{S_Y^2/\sigma_Y^2} \sim F_{m-1,\,n-1}

Independent samples of sizes $m$ and $n$ from normal populations.

Sampling distributions

Chi-square

Degrees of freedom $\nu \in \{1,2,\dots\}$ , written $\chi^2_\nu$ . Same law as Gamma with shape $\nu/2$ and rate $1/2$ .

pdf: $\displaystyle \frac{1}{2^{\nu/2}\Gamma(\nu/2)} x^{\nu/2 - 1} e^{-x/2}$ for $x > 0$
Mean: $\displaystyle \nu$
Variance: $\displaystyle 2\nu$
mgf: $\displaystyle (1 - 2t)^{-\nu/2}, \; t < \tfrac{1}{2}$

Student t

Degrees of freedom $\nu > 0$ , written $t_\nu$ . Symmetric about $0$ with heavier tails than $N(0,1)$ ; $t_\nu \to N(0,1)$ in distribution as $\nu \to \infty$ . $\nu = 1$ is the Cauchy law (no mean).

pdf: $\displaystyle \frac{\Gamma\left(\frac{\nu+1}{2}\right)}{\sqrt{\nu\pi}\,\Gamma\left(\frac{\nu}{2}\right)} \left(1 + \frac{x^2}{\nu}\right)^{-\frac{\nu+1}{2}}$ for $x \in \mathbb{R}$
Mean: $\displaystyle 0 \quad (\nu > 1)$
Variance: $\displaystyle \frac{\nu}{\nu - 2} \quad (\nu > 2)$
mgf: Does not exist: $E[e^{tX}] = \infty$ for every $t \ne 0$ , and only moments of order below $\nu$ are finite.

Degrees of freedom $d_1, d_2 > 0$ , written $F_{d_1, d_2}$ . Here $B(\cdot,\cdot)$ is the Beta function.

pdf: $\displaystyle \frac{1}{B\left(\frac{d_1}{2}, \frac{d_2}{2}\right)} \left(\frac{d_1}{d_2}\right)^{d_1/2} x^{d_1/2 - 1} \left(1 + \frac{d_1}{d_2}x\right)^{-(d_1+d_2)/2}$ for $x > 0$
Mean: $\displaystyle \frac{d_2}{d_2 - 2} \quad (d_2 > 2)$
Variance: $\displaystyle \frac{2d_2^2(d_1 + d_2 - 2)}{d_1(d_2-2)^2(d_2-4)} \quad (d_2 > 4)$
mgf: Does not exist: $E[e^{tX}] = \infty$ for every $t > 0$ because the right tail is polynomial.

Order statistic distributions

f_{X_{(k)}}(x) = \frac{n!}{(k-1)!\,(n-k)!}\,F(x)^{k-1}\big[1 - F(x)\big]^{n-k} f(x), \qquad F_{X_{(n)}}(x) = F(x)^n, \qquad F_{X_{(1)}}(x) = 1 - \big[1 - F(x)\big]^n

$X_1,\dots,X_n$ iid with cdf $F$ , sorted as $X_{(1)} \le \cdots \le X_{(n)}$ , and $k \in \{1,\dots,n\}$ . The density formula needs a pdf $f$ ; the max and min cdfs hold for any iid sample. Example: for $\mathrm{Unif}(0,\theta)$ , $F_{X_{(n)}}(x) = (x/\theta)^n$ for $0 \le x \le \theta$ .

Unit 2

Sufficiency, completeness, and exponential families

Unit 3

Bias, risk, consistency, and point estimation

Unit 4

Likelihood, score, and Fisher information

Unit 5

Pivots and confidence sets

Unit 6

Loss, risk, and introductory decision theory

This sheet was still loading. Close this and print again once every unit shows its formulas.