Multiple linear regression

Types of predictors

Prof. Maria Tackett

Jan 28, 2025

Announcements

  • Lab 02 due TODAY at 11:59pm

  • HW 01 due Thursday, January 30 at 11:59pm

Topics

  • Categorical predictors

  • Centering quantitative predictors

  • Standardizing quantitative predictors

  • Interaction terms

Computing setup

# load packages
library(tidyverse)
library(tidymodels)
library(openintro)
library(patchwork)
library(knitr)
library(kableExtra)
library(viridis) #adjust color palette

# set default theme and larger font size for ggplot2
ggplot2::theme_set(ggplot2::theme_minimal(base_size = 16))

Data: Peer-to-peer lender

Today’s data is a sample of 50 loans made through a peer-to-peer lending club. The data is in the loan50 data frame in the openintro R package.

# A tibble: 50 × 4
   annual_income_th debt_to_income verified_income interest_rate
              <dbl>          <dbl> <fct>                   <dbl>
 1             59           0.558  Not Verified            10.9 
 2             60           1.31   Not Verified             9.92
 3             75           1.06   Verified                26.3 
 4             75           0.574  Not Verified             9.92
 5            254           0.238  Not Verified             9.43
 6             67           1.08   Source Verified          9.92
 7             28.8         0.0997 Source Verified         17.1 
 8             80           0.351  Not Verified             6.08
 9             34           0.698  Not Verified             7.97
10             80           0.167  Source Verified         12.6 
# ℹ 40 more rows

Variables

Predictors:

  • annual_income_th: Annual income (in $1000s)
  • debt_to_income: Debt-to-income ratio, i.e. the percentage of a borrower’s total debt divided by their total income
  • verified_income: Whether borrower’s income source and amount have been verified (Not Verified, Source Verified, Verified)

Response: interest_rate: Interest rate for the loan

Response vs. predictors

Goal: Use these predictors in a single model to understand variability in interest rate.

Model fit in R

int_fit <- lm(interest_rate ~ debt_to_income + verified_income  + annual_income_th,
              data = loan50)

tidy(int_fit) |>
  kable(digits = 3)
term estimate std.error statistic p.value
(Intercept) 10.726 1.507 7.116 0.000
debt_to_income 0.671 0.676 0.993 0.326
verified_incomeSource Verified 2.211 1.399 1.581 0.121
verified_incomeVerified 6.880 1.801 3.820 0.000
annual_income_th -0.021 0.011 -1.804 0.078

Categorical predictors

Matrix form of multiple linear regression

[y1⋮yn]⏟y=[1x11…x1p⋮⋮⋱⋮1xn1…xnp]⏟X[β0β1⋮βp]⏟β+[ϵ1⋮ϵn]⏟ϵ

How might we include a categorical predictor with k levels in the design matrix, X ?

Indicator variables

  • Suppose there is a categorical variable with k levels

  • We can make k indicator variables from the data - one indicator for each level

  • An indicator (dummy) variable takes values 1 or 0

    • 1 if the observation belongs to that level

    • 0 if the observation does not belong to that level

Indicator variables

Suppose we want to predict the amount of sleep a Duke student gets based on whether they are in Pratt (Pratt Yes/ No are the only two options). Consider the model

Sleepi=β0+β11(Pratti=Yes)+β21(Pratti=No)

  • Write out the design matrix for this hypothesized linear model.

  • Demonstrate that the design matrix is not of full column rank (that is, affirmatively provide one of the columns in terms of the others).

  • Use this intuition to explain why when we include categorical predictors, we cannot include both indicators for every level of the variable and an intercept.

Indicator variables for verified_income

loan50 <- loan50 |>
  mutate(
    not_verified = factor(if_else(verified_income == "Not Verified", 1, 0)),
    source_verified = factor(if_else(verified_income == "Source Verified", 1, 0)),
    verified = factor(if_else(verified_income == "Verified", 1, 0))
  )
# A tibble: 3 × 4
  verified_income not_verified source_verified verified
  <fct>           <fct>        <fct>           <fct>   
1 Not Verified    1            0               0       
2 Verified        0            0               1       
3 Source Verified 0            1               0

Indicator variables in the model

Given a categorical predictor with k levels…

  • Use k−1 indicator variables in the model
  • The baseline is the category that doesn’t have a term in the model
    • This is also called the reference level
  • The coefficients of the indicator variables in the model are interpreted as the expected change in the response compared to the baseline, holding all other variables constant.

Application exercise

📋 https://sta221-sp25.netlify.app/ae/ae-02-mlr.html

Complete Part 1

Interpreting verified_income

term estimate std.error statistic p.value conf.low conf.high
(Intercept) 10.726 1.507 7.116 0.000 7.690 13.762
debt_to_income 0.671 0.676 0.993 0.326 -0.690 2.033
verified_incomeSource Verified 2.211 1.399 1.581 0.121 -0.606 5.028
verified_incomeVerified 6.880 1.801 3.820 0.000 3.253 10.508
annual_income_th -0.021 0.011 -1.804 0.078 -0.043 0.002
  • The baseline level is Not verified.
  • People with source verified income are expected to take a loan with an interest rate that is 2.211% higher, on average, than the rate on loans to those whose income is not verified, holding all else constant.

Centering

  • Centering a quantitative predictor means shifting every value by some constant C

  • One common type of centering is mean-centering, in which every value of a predictor is shifted by its mean

  • Only quantitative predictors are centered

  • Center all quantitative predictors in the model for ease of interpretation

What is one reason one might want to center the quantitative predictors? What is are the units of centered variables?

Centering

Use the scale() function with center = TRUE and scale = FALSE to mean-center variables

loan50 <- loan50 |>
  mutate(debt_to_inc_cent = scale(debt_to_income, center = TRUE, scale = FALSE), 
         annual_inc_cent = scale(annual_income_th, center = TRUE, scale = FALSE))

lm(interest_rate ~ debt_to_inc_cent + verified_income + annual_inc_cent, data = loan50) |> 
  tidy() |> kable(digits = 3)
term estimate std.error statistic p.value
(Intercept) 9.444 0.977 9.663 0.000
debt_to_inc_cent 0.671 0.676 0.993 0.326
verified_incomeSource Verified 2.211 1.399 1.581 0.121
verified_incomeVerified 6.880 1.801 3.820 0.000
annual_inc_cent -0.021 0.011 -1.804 0.078

Centering

Term Original Model Centered Model
(Intercept) 10.726 9.444
debt_to_income 0.671 0.671
verified_incomeSource Verified 2.211 2.211
verified_incomeVerified 6.880 6.880
annual_income_th -0.021 -0.021

How has the model changed? How has the model remained the same?

Standardizing

  • Standardizing a quantitative predictor mean shifting every value by the mean and dividing by the standard deviation of that variable

  • Only quantitative predictors are standardized

  • Standardize all quantitative predictors in the model for ease of interpretation

What is one reason one might want to standardize the quantitative predictors? What is are the units of standardized variables?

Standardizing

Use the scale() function with center = TRUE and scale = TRUE to standardized variables

loan50 <- loan50 |>
  mutate(debt_to_inc_std = scale(debt_to_income, center = TRUE, scale = TRUE), 
         annual_inc_std = scale(annual_income_th, center = TRUE, scale = TRUE))

lm(interest_rate ~ debt_to_inc_std + verified_income + annual_inc_std, data = loan50) |>
  tidy() |> kable(digits = 3)
term estimate std.error statistic p.value
(Intercept) 9.444 0.977 9.663 0.000
debt_to_inc_std 0.643 0.648 0.993 0.326
verified_incomeSource Verified 2.211 1.399 1.581 0.121
verified_incomeVerified 6.880 1.801 3.820 0.000
annual_inc_std -1.180 0.654 -1.804 0.078

Standardizing

Term Original Model Standardized Model
(Intercept) 10.726 9.444
debt_to_income 0.671 0.643
verified_incomeSource Verified 2.211 2.211
verified_incomeVerified 6.880 6.880
annual_income_th -0.021 -1.180

How has the model changed? How has the model remained the same?

Interaction terms

Interaction terms

  • Sometimes the relationship between a predictor variable and the response depends on the value of another predictor variable.
  • This is an interaction effect.
  • To account for this, we can include interaction terms in the model.

Interest rate vs. annual income

The lines are not parallel indicating there is a potential interaction effect. The slope of annual income differs based on the income verification.

Application exercise

📋 https://sta221-sp25.netlify.app/ae/ae-02-mlr.html

Complete Part 2

Interaction term in model

int_fit_2 <- lm(interest_rate ~ debt_to_income + verified_income + annual_income_th + verified_income * annual_income_th,
      data = loan50)
term estimate std.error statistic p.value
(Intercept) 9.560 2.034 4.700 0.000
debt_to_income 0.691 0.685 1.009 0.319
verified_incomeSource Verified 3.577 2.539 1.409 0.166
verified_incomeVerified 9.923 3.654 2.716 0.009
annual_income_th -0.007 0.020 -0.341 0.735
verified_incomeSource Verified:annual_income_th -0.016 0.026 -0.643 0.523
verified_incomeVerified:annual_income_th -0.032 0.033 -0.979 0.333

Interpreting interaction terms

  • What the interaction means: The effect of annual income on the interest rate differs by -0.016 when the income is source verified compared to when it is not verified, holding all else constant.
  • Interpreting annual_income for source verified: If the income is source verified, we expect the interest rate to decrease by 0.023% (-0.007 + -0.016) for each additional thousand dollars in annual income, holding all else constant.

Recap

  • Interpreted categorical predictors

  • Explored by k−1 indicators are included in a model

  • Fit and interpreted models with centered and standardized variables

  • Interpreted interaction terms

Next class

  • Model comparison for multiple linear regression

🔗 STA 221 - Spring 2025

1 / 30
Multiple linear regression Types of predictors Prof. Maria Tackett Jan 28, 2025

  1. Slides

  2. Tools

  3. Close
  • Multiple linear regression
  • Announcements
  • Topics
  • Computing setup
  • Data: Peer-to-peer lender
  • Variables
  • Response vs. predictors
  • Model fit in R
  • Categorical predictors
  • Matrix form of multiple linear regression
  • Indicator variables
  • Indicator variables
  • Indicator variables for verified_income
  • Indicator variables in the model
  • Application exercise
  • Interpreting verified_income
  • Centering
  • Centering
  • Centering
  • Standardizing
  • Standardizing
  • Standardizing
  • Interaction terms
  • Interaction terms
  • Interest rate vs. annual income
  • Application exercise
  • Interaction term in model
  • Interpreting interaction terms
  • Recap
  • Next class
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • r Scroll View Mode
  • b Toggle Chalkboard
  • c Toggle Notes Canvas
  • d Download Drawings
  • ? Keyboard Help