Supervised learning: linear regression
Visual explanation
Imagine a 2D scatterplot with different data points
Linear regression aims to find a “best fit” line that minimizes the vertical distance between the line and the data points
The vertical distance is known as a residual (difference between the actual y-value of a data point and the predicted y-value on the regression line)
Modeling NFL score differential
Interested in modeling a team’s score differential
nfl_teams_data <- nfl_teams_data %>%
mutate(score_diff =
points_scored - points_allowed)
nfl_teams_data |>
ggplot(aes(x = score_diff)) +
geom_histogram(color = "black",
fill = "darkblue",
alpha = 0.3) +
labs(x = "Score differential", y = "Count")
Relationship between score differential and passing performance
offense_ave_epa_pass is the team’s average expected points added (EPA) per pass attempt
nfl_teams_data |>
ggplot(aes(x = offense_ave_epa_pass,
y = score_diff)) +
geom_point(alpha = 0.5) +
labs(x = "EPA gained per pass attempt",
y = "Score differential")We fit linear regression models using lm(), formula is input as: response ~ predictor
simple_lm <- lm(score_diff ~ offense_ave_epa_pass, data = nfl_teams_data) 
Plot observed values against predictions
Useful diagnostic (for any type of model, not just linear regression!)
nfl_teams_data |>
mutate(pred = predict(simple_lm)) |>
ggplot(aes(x = pred, y = score_diff)) +
geom_point(alpha = 0.5, size = 1.5) +
geom_abline(slope = 1, intercept = 0,
linetype = "dashed",
color = "orangered",
linewidth = 2) A “perfect” model will follow diagonal
Plot observed values against predictions
Augment the data with model output using the broom package
nfl_teams_data <- simple_lm |>
augment(nfl_teams_data)
nfl_teams_data |>
ggplot(aes(x = .fitted, y = score_diff)) +
geom_point(alpha = 0.5, size = 1.5) +
geom_abline(slope = 1, intercept = 0, color = "orangered",
linetype = "dashed", linewidth = 2)augment() adds various columns from model fit we can use in plotting for model diagnostics
Plot residuals against predicted values
nfl_teams_data |>
ggplot(aes(x = .fitted, y = .resid)) +
geom_point(alpha = 0.5, size = 1.5) +
geom_hline(yintercept = 0, linetype = "dashed",
color = "orangered", linewidth = 2) +
# plot the residual mean
geom_smooth(se = TRUE, fill = "skyblue2")Two things to look for:
Any trend around horizontal reference line? Violates linearity (may have to additional covariates, polynomial terms, splines, etc.)
Unequal vertical spread? Violates homoskedasticity (consider using the sandwich estimator)
What about the normality assumption?
OLS doesn’t care about this assumption, it’s just estimating coefficients!
BUT in terms of inference, a violation of the normality of errors assumption:
May result in incorrect inferences in small samples
Usually does not have a big impact on inferences in larger samples (Weisberg 2014)
We can detect whether the errors are (roughly) normally distributed by Q-Q plot on the standardized residuals, which are the residuals divided by their estimated standard deviation
\[ \displaystyle \hat{\epsilon}_{\text{standardized}} = \frac{\hat{\epsilon}}{\hat{\sigma}} \]
nfl_teams_data |>
ggplot(aes(sample = .std.resid)) +
geom_qq_line(distribution = stats::qnorm) +
geom_qq(distribution = stats::qnorm) 
Comparing observed values and predictions
Similar to before, we can plot the actual and predicted ridership
To reiterate, all we’re doing is changing the intercept of our regression line by including a categorical variable
nfl_teams_subset |>
mutate(team_pred = predict(team_lm)) |>
ggplot(aes(x = score_diff, y = team_pred)) +
geom_point(size = 1.5, alpha = 0.5) +
facet_wrap(~ team, ncol = 3) +
labs(x = "Actual score differential",
y = "Predicted score differential")
Visualizing interactions
nfl_teams_subset |>
ggplot(aes(x = offense_ave_epa_pass, y = score_diff, color = team)) +
geom_point(size = 2, alpha = 0.6) +
geom_smooth(method = "lm", se = FALSE, linewidth = 2) +
ggthemes::scale_color_colorblind()
Regression confidence intervals
The layer geom_smooth() displays confidence intervals for the regression line
# predict(simple_lm, interval = "confidence")
lm_plot <- nfl_teams_subset |>
ggplot(aes(x = offense_ave_epa_pass, y = score_diff)) +
geom_point(alpha = 0.5) +
geom_smooth(method = "lm")
lm_plot
Confidence intervals versus prediction intervals
The standard error for predicting an individual response will always be larger than for predicting a mean response
Prediction intervals will always be wider than confidence intervals
# predict(simple_lm, interval = "prediction")
lm_plot +
geom_ribbon(
data = augment(simple_lm, interval = "prediction"),
aes(ymin = .lower, ymax = .upper),
color = "red", fill = NA
)