STA 221 - Spring 2025 – Inference for regression

term	estimate	std.error	statistic
(Intercept)	19.332	2.984	6.478
enrollment_th	0.780	0.110	7.074
typePublic	-13.226	3.153	-4.195

Statistical inference

Statistical inference provides methods and tools so we can use the single observed sample to make valid statements (inferences) about the population it comes from
For our inferences to be valid, the sample should be representative (ideally random) of the population we’re interested in

Image source: Eugene Morgan © Penn State

Inference for linear regression

Inference based on ANOVA
- Hypothesis test for the statistical significance of the overall regression model
- Hypothesis test for a subset of coefficients
Inference for a single coefficient $β_{j}$ (today’s focus)
- Hypothesis test for a coefficient $β_{j}$
- Confidence interval for a coefficient $β_{j}$

Linear regression model

$\begin{aligned} y & = Model + Error \\ = f (X) + ϵ \\ = E (y | X) + ϵ \\ = X β + ϵ \end{aligned}$

We have discussed multiple ways to find the least squares estimates of $β = [\begin{matrix} β_{0} \\ β_{1} \end{matrix}]$
- None of these approaches depend on the distribution of $ϵ$
Now we will use statistical inference to draw conclusions about $β$ that depend on particular assumptions about the distribution of $ϵ$

Linear regression model

$\begin{array}{r} Y = X β + ϵ, ϵ \sim N (0, σ_{ϵ}^{2} I) \end{array}$

such that the errors are independent and normally distributed.

Independent: Knowing the error term for one observation doesn’t tell us about the error term for another observation
Normally distributed: The distribution follows a particular mathematical model that is unimodal and symmetric

Describing random phenomena

There is some uncertainty in the error terms (and thus the response variable), so we use mathematical models to describe that uncertainty.
Some terminology:
- Sample space: Set of all possible outcomes
- Random variable: Function (mapping) from the sample space onto real numbers
- Event: Subset of the sample space, i.e., a set of possible outcomes (possible values the random variable can take)
- Probability density function: Mathematical function that produces probability of occurrences for events in the sample space for a continuous random variable

Distribution of error terms

The error terms follow a (multivariate) normal distribution with mean $0$ and variance $σ^{2} I$

$f (ϵ) = \frac{1}{(2 π)^{n / 2} | σ_{ϵ}^{2} I |^{1 / 2}} \exp {- \frac{1}{2} (ϵ - 0)^{T} (σ_{ϵ}^{2} I)^{- 1} (ϵ - 0)}$

Visualizing distribution of $y | X$

$y | X \sim N (X β, σ_{ϵ}^{2} I)$

Image source: Introduction to the Practice of Statistics (5th ed)

Expected value

Let $z = [\begin{matrix} z_{1} \\ ⋮ \\ z_{p} \end{matrix}]$ be a $p \times 1$ vector of random variables.

Then $E (z) = E [\begin{matrix} z_{1} \\ ⋮ \\ z_{p} \end{matrix}] = [\begin{matrix} E (z_{1}) \\ ⋮ \\ E (z_{p}) \end{matrix}]$

Expected value

Let $A$ be an $n \times p$ matrix of constants, $C$ a $n \times 1$ vector of constants, and $z$ a $p \times 1$ vector of random variables. Then

$E (Az) = A E (z)$

$E (Az + C) = E (Az) + E (C) = A E (z) + C$

Expected value of the response

Show $E (y | X) = X β$

Variance

Let $z = [\begin{matrix} z_{1} \\ ⋮ \\ z_{p} \end{matrix}]$ be a $p \times 1$ vector of random variables.

Then $V a r (z) = [\begin{matrix} V a r (z_{1}) & C o v (z_{1}, z_{2}) & \dots & C o v (z_{1}, z_{p}) \\ C o v (z_{2}, z_{1}) & V a r (z_{2}) & \dots & C o v (z_{2}, z_{p}) \\ ⋮ & ⋮ & \dots & \cdot \\ C o v (z_{p}, z_{1}) & C o v (z_{p}, z_{2}) & \dots & V a r (z_{p}) \end{matrix}]$

Variance

Let $A$ be an $n \times p$ matrix of constants and $z$ a $p \times 1$ vector of random variables. Then

$V a r (z) = E [(z - E (z)) (z - E (z))^{T}]$

$\begin{aligned} V a r (Az) & = E [(Az - E (Az)) (Az - E (Az))^{T}] \\ = A V a r (z) A^{T} \end{aligned}$

Variance of the response

Show

$V a r (y | X) = σ_{ϵ}^{2} I$

Linear transformation of normal random variable

Suppose $z$ is a (multivariate) normal random variable such that $z \sim N (μ, Σ)$

A linear transformation of $z$ is also multivariate normal, such that

$A z + B \sim N (A μ + B, A Σ A^{T})$

Explain why $y | X$ is normally distributed.

Recap

Introduced statistical inference in the context of regression
Described the assumptions for regression
Connected the distribution of residuals and inferential procedures

Next class

Confidence intervals for ${\hat{β}}_{j}$
Hypothesis testing based on ANOVA
See Prepare for Lecture 09

Inference for regression

Announcements

Poll: Office hours availability

Topics

Computing setup

Data: NCAA Football expenditures

Univariate EDA

Bivariate EDA

Regression model

From sample to population

Inference for regression

Statistical inference

Inference for linear regression

Linear regression model

Linear regression model

Describing random phenomena

Distribution of error terms

Visualizing distribution of $y | X$

Expected value

Expected value

Expected value of the response

Variance

Variance

Variance of the response

Linear transformation of normal random variable

Recap

Next class