SLR: Matrix representation

Announcements

• Lab 01 due on TODAY at 11:59pm

  • Push work to GitHub repo

  • Submit final PDF on Gradescope + mark pages for each question

• HW 01 will be assigned on Thursday

Topics

• Application exercise on model assessment
• Matrix representation of simple linear regression
  • Model form
  • Least square estimate
  • Predicted (fitted) values
  • Residuals

Model assessment

Two statistics

• Root mean square error, RMSE: A measure of the average error (average difference between observed and predicted values of the outcome)

  $$RMSE=\sqrt{\frac{\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2}{n}}=\sqrt{\frac{\sum _{i=1}^n{e}_i^2}{n}}$$

• R-squared, $R^2$ : Percentage of variability in the outcome explained by the regression model (in the context of SLR, the predictor)

$$R^2=\frac{SSM}{SST}=1-\frac{SSR}{SST}$$

Application exercise

📋 sta221-sp25.netlify.app/ae/ae-01-model-assessment.html

Open ae-01 from last class. Complete Part 2.

Matrix representation of simple linear regression

SLR: Statistical model (population)

When we have a quantitative response, $Y$, and a single quantitative predictor, $X$, we can use a simple linear regression model to describe the relationship between $Y$ and $X$.

$$Y={\beta }_0+{\beta }_{1}X+ϵ$$

• ${\beta }_1$: Population (true) slope of the relationship between $X$ and $Y$
• ${\beta }_0$: Population (true) intercept of the relationship between $X$ and $Y$
• $ϵ$: Error terms centered at 0 with variance ${\sigma }_{ϵ}^2$

SLR in matrix form

The simple linear regression model can be represented using vectors and matrices as

$$\mathbf{y}=\mathbf{X}\mathit{\beta }+\mathit{ϵ}$$

• $\mathbf{y}$ : Vector of responses

• $\mathbf{X}$: Design matrix (columns for predictors + intercept)

• $\mathit{\beta }$: Vector of model coefficients

• $\mathit{ϵ}$: Vector of error terms centered at $\mathbf{0}$ with variance ${\sigma }_{ϵ}^2\mathbf{I}$

SLR in matrix form

$$\underset{\mathbf{y}}{\underbrace{\left[\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right]}}=\underset{\mathbf{X}}{\underbrace{\left[\begin{array}{cc}1& x_1\\ ⋮& ⋮\\ 1& x_n\end{array}\right]}}\underset{\mathit{\beta }}{\underbrace{\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right]}}+\underset{\mathit{ϵ}}{\underbrace{\left[\begin{array}{c}{ϵ}_1\\ ⋮\\ {ϵ}_n\end{array}\right]}}$$

What are the dimensions of $\mathbf{y}$, $\mathbf{X}$, $\mathit{\beta }$, and $\mathit{ϵ}$?

Derive least squares estimator for $\mathit{\beta }$

Goal: Find estimator $\hat{\mathit{\beta }}=\left[\begin{array}{c}{\hat{\beta }}_0\\ {\hat{\beta }}_1\end{array}\right]$ that minimizes the sum of squared errors $$\sum _{i=1}^n{ϵ}_i^2={ϵ}^{\mathsf{T}}ϵ=(\mathbf{y}-\mathbf{X}\mathit{\beta }{)}^{\mathsf{T}}(\mathbf{y}-\mathbf{X}\mathit{\beta })$$

Gradient

Let $\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_k\end{array}\right]$be a $k\times 1$ vector and $f(\mathbf{x})$ be a function of $\mathbf{x}$.

. . .

Then ${\mathrm{\nabla }}_{\mathbf{x}}f$, the gradient of $f$ with respect to $\mathbf{x}$ is

$${\mathrm{\nabla }}_{\mathbf{x}}f=\left[\begin{array}{c}\frac{\partial f}{\partial x_1}\\ \frac{\partial f}{\partial x_2}\\ ⋮\\ \frac{\partial f}{\partial x_k}\end{array}\right]$$

Property 1

Let $\mathbf{x}$ be a $k\times 1$ vector and $\mathbf{z}$ be a $k\times 1$ vector, such that $\mathbf{z}$ is not a function of $\mathbf{x}$ .

The gradient of ${\mathbf{x}}^{\mathsf{T}}\mathbf{z}$ with respect to $\mathbf{x}$ is

$${\mathrm{\nabla }}_{\mathbf{x}}{\mathbf{x}}^{\mathsf{T}}\mathbf{z}=\mathbf{z}$$

Side note: Property 1

$$\begin{array}{rl}{\mathbf{x}}^{\mathsf{T}}\mathbf{z}& =\left[\begin{array}{cccc}{x}_1& x_2& \dots & x_k\end{array}\right]\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_k\end{array}\right]\\ & =x_1{z}_1+x_2{z}_2+\cdots +x_k{z}_k\\ & =\sum _{i=1}^k{x}_i{z}_i\end{array}$$

Side note: Property 1

$${\mathrm{\nabla }}_{\mathbf{x}}{\mathbf{x}}^{\mathsf{T}}\mathbf{z}=\left[\begin{array}{c}\frac{\partial {\mathbf{x}}^{\mathsf{T}}\mathbf{z}}{\partial x_1}\\ \frac{\partial {\mathbf{x}}^{\mathsf{T}}\mathbf{z}}{\partial x_2}\\ ⋮\\ \frac{\partial {\mathbf{x}}^{\mathsf{T}}\mathbf{z}}{\partial x_k}\end{array}\right]=\left[\begin{array}{c}\frac{\partial }{\partial x_1}(x_1{z}_1+x_2{z}_2+\cdots +x_k{z}_k)\\ \frac{\partial }{\partial x_2}(x_1{z}_1+x_2{z}_2+\cdots +x_k{z}_k)\\ ⋮\\ \frac{\partial }{\partial x_k}(x_1{z}_1+x_2{z}_2+\cdots +x_k{z}_k)\end{array}\right]=\left[\begin{array}{c}{z}_1\\ z_2\\ ⋮\\ z_k\end{array}\right]=\mathbf{z}$$

Property 2

Let $\mathbf{x}$ be a $k\times 1$ vector and $\mathbf{A}$ be a $k\times k$ matrix, such that $\mathbf{A}$ is not a function of $\mathbf{x}$ .

Then the gradient of ${\mathbf{x}}^{\mathsf{T}}\mathbf{A}\mathbf{x}$ with respect to $\mathbf{x}$ is

$${\mathrm{\nabla }}_{\mathbf{x}}{\mathbf{x}}^{\mathsf{T}}\mathbf{A}\mathbf{x}=(\mathbf{A}\mathbf{x}+{\mathbf{A}}^{\mathsf{T}}\mathbf{x})=(\mathbf{A}+{\mathbf{A}}^{\mathsf{T}})\mathbf{x}$$

If $\mathbf{A}$ is symmetric, then

$$(\mathbf{A}+{\mathbf{A}}^{\mathsf{T}})\mathbf{x}=2\mathbf{A}\mathbf{x}$$

Proof in HW 01.

Derive least squares estimator

Find $\hat{\mathit{\beta }}$ that minimizes

$$\begin{array}{rl}{\mathit{ϵ}}^{\mathsf{T}}\mathit{ϵ}& =(\mathbf{y}-\mathbf{X}\mathit{\beta }{)}^{\mathsf{T}}(\mathbf{y}-\mathbf{X}\mathit{\beta })\\ & =({\mathbf{y}}^{\mathsf{T}}-{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}})(\mathbf{y}-\mathbf{X}\mathit{\beta })\\ & ={\mathbf{y}}^{\mathsf{T}}\mathbf{y}-{\mathbf{y}}^{\mathsf{T}}\mathbf{X}\mathit{\beta }-{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\mathit{\beta }\\ & ={\mathbf{y}}^{\mathsf{T}}\mathbf{y}-2{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\mathit{\beta }\end{array}$$

Derive least squares estimator

$$\begin{array}{rl}{\mathrm{\nabla }}_{\beta }{\mathit{ϵ}}^{\mathsf{T}}\mathit{ϵ}& ={\mathrm{\nabla }}_{\mathit{\beta }}({\mathbf{y}}^{\mathsf{T}}\mathbf{y}-2{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+{\mathit{\beta }}^{\mathsf{T}}{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\mathit{\beta })\\ & =-2{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+2{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\mathit{\beta }\end{array}$$

Find $\hat{\mathit{\beta }}$ that satisfies

$$-2{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+2{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\hat{\mathit{\beta }}=\mathbf{0}$$

$$\hat{\mathit{\beta }}=({\mathbf{X}}^{\mathsf{T}}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathsf{T}}\mathbf{y}$$

Did we find a minimum?

Hessian matrix

The Hessian matrix, ${\mathrm{\nabla }}_{\mathbf{x}}^{2}f$ is a $k\times k$ matrix of partial second derivatives

$${\mathrm{\nabla }}_{\mathbf{x}}^{2}f=\left[\begin{array}{cccc}\frac{{\partial }^{2}f}{\partial x_1^2}& \frac{{\partial }^{2}f}{\partial x_1\partial x_2}& \dots & \frac{{\partial }^{2}f}{\partial x_1\partial x_k}\\ \frac{{\partial }^{2}f}{\partial x_2\partial x_1}& \frac{{\partial }^{2}f}{\partial x_2^2}& \dots & \frac{{\partial }^{2}f}{\partial x_2\partial x_k}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}f}{\partial x_k\partial x_1}& \frac{{\partial }^{2}f}{\partial x_k\partial x_2}& \dots & \frac{{\partial }^{2}f}{\partial x_k^2}\end{array}\right]$$

Using the Hessian matrix

If the Hessian matrix is…

• positive-definite, then we have found a minimum.

• negative-definite, then we have found a maximum.

• neither positive or negative-definite, then we have found a saddle point

Did we find a minimum?

$$\begin{array}{rl}{\mathrm{\nabla }}_{\mathit{\beta }}^2{\mathit{ϵ}}^{\mathsf{T}}\mathit{ϵ}& ={\mathrm{\nabla }}_{\mathit{\beta }}(-2{\mathbf{X}}^{\mathsf{T}}\mathbf{y}+2{\mathbf{X}}^{\mathsf{T}}\mathbf{X}\mathit{\beta })\\ & =-2{\mathrm{\nabla }}_{\mathit{\beta }}({\mathbf{X}}^{\mathsf{T}}\mathbf{y})+2{\mathrm{\nabla }}_{\mathit{\beta }}({\mathbf{X}}^{\mathsf{T}}\mathbf{X}\beta )\\ & \propto {\mathbf{X}}^{\mathsf{T}}\mathbf{X}\end{array}$$

Show that ${\mathbf{X}}^{\mathsf{T}}\mathbf{X}$ is positive definite in HW 01.

Predicted values and residuals

Predicted (fitted) values

Now that we have $\hat{\mathit{\beta }}$, let’s predict values of $\mathbf{y}$ using the model

$$\hat{\mathbf{y}}=\mathbf{X}\hat{\mathit{\beta }}=\underset{\mathbf{H}}{\underbrace{\mathbf{X}({\mathbf{X}}^{\mathsf{T}}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathsf{T}}}}\mathbf{y}=\mathbf{H}\mathbf{y}$$

. . .

Hat matrix: $\mathbf{H}=\mathbf{X}({\mathbf{X}}^{\mathsf{T}}\mathbf{X}{)}^{-1}{\mathbf{X}}^{\mathsf{T}}$

. . .

• $\mathbf{H}$ is an $n\times n$ matrix
• Maps vector of observed values $\mathbf{y}$ to a vector of fitted values $\hat{\mathbf{y}}$
• It is only a function of $\mathbf{X}$ not $\mathbf{y}$

Residuals

Recall that the residuals are the difference between the observed and predicted values

$$\begin{array}{rl}\mathbf{e}& =\mathbf{y}-\hat{\mathbf{y}}\\ & =\mathbf{y}-\mathbf{X}\hat{\mathit{\beta }}\\ & =\mathbf{y}-\mathbf{H}\mathbf{y}\\ \mathbf{e}& =(\mathbf{I}-\mathbf{H})\mathbf{y}\end{array}$$

Recap

• Introduced matrix representation for simple linear regression

  • Model form
  • Least square estimate
  • Predicted (fitted) values
  • Residuals

For next class

• Complete Prepare for Lecture 05 - SLR: matrix representation cont’d