Linear Regression — The Math, Not Just the API

What `LinearRegression().fit()` actually does, and the assumptions that make it valid. FIND_VIDEO: search 'linear regression OLS assumptions' — recommended channel: StatQuest / 3Blue1Brown. Aim for 11 min or under.

27 minutesVideo LessonPDF notes

Watch Linear Regression — The Math, Not Just the API on YouTube

🎯 Free Guest Mode: You are learning for free. Sign in to save your completion progress and quiz answers.

Ready to continue?

Mark this lesson as complete when you're ready to proceed.

Key moments

Core Regression Steps — Regression analysis involves fitting a line, quantifying performance, and testing statistical significance.
Least Squares Method — The best-fit line minimizes the Sum of Squared Residuals (SSR), which are the errors between data and the line.
Model Parameters — The fitted line is defined by the estimated slope and Y-intercept, which determine its predictive power.
Baseline Variation — SS Mean establishes the total variation in Y that must be explained, calculated around the average Y value.
Residual Variation — SS Fit quantifies the unexplained variation remaining after the least squares line has been fitted to the data.
R-Squared Formula — R-squared is calculated as the ratio of explained variation to total variation, showing goodness-of-fit.
Boundary Cases — R-squared ranges from 0% (no fit) to 100% (perfect fit) and applies universally across model types.
Multiple Predictors — Linear regression extends to multiple variables by fitting a plane or hyperplane in higher dimensions.

PDF notes

Bottom line: You will be able to mathematically derive and calculate the core components of a linear regression model, moving beyond API calls to understand the underlying statistical mechanics. You will master the Least Squares Method for fitting the line and quantify model performance using the R-squared metric based on sums of squares.

This lesson on Linear Regression — The Math, Not Just the API is hands-on and example-driven. You will be able to mathematically derive and calculate the core components of a linear regression model, moving beyond API calls to understand the underlying statistical mechanics. You will master the Least Squares Method for fitting the line and quantify model performance using the R-squared metric based on sums of squares.

What You'll Be Able To Do

Calculate the residual error for any given data point relative to a fitted line.
Apply the Least Squares Method to determine the optimal slope and intercept parameters.
Define and calculate the Sum of Squared Residuals (SS Fit) and the total variation (SS Mean).
Derive the R-squared value using the ratio of explained variation to total variation.
Generalize the concept of Sums of Squares to evaluate complex non-linear models.

Detailed Concept Walkthrough

1. Least Squares Fitting and Residuals

The Least Squares Method (LSM) finds the line that minimizes the total vertical distance between the line and all data points. This distance, called the residual, represents the error of the model's prediction.

Mechanism: LSM iteratively adjusts the slope and intercept until the Sum of Squared Residuals (SSR or SS Fit) reaches its absolute minimum. Squaring the residuals ensures positive errors do not cancel out negative errors.
Under the Hood: The optimal parameters (slope $m$ and intercept $b$) are found by taking the partial derivatives of the SSR function with respect to $m$ and $b$, setting them to zero, and solving the resulting system of equations.
Best Practice: Always visualize the residuals (e.g., in a scatter plot of residuals vs. predicted values) to check for non-linear patterns, which indicate the linear model is inappropriate.

import numpy as np

# Assume X and Y are data arrays, m and b are parameters
def calculate_ssr(X, Y, m, b):
    # Calculate predicted Y values
    Y_pred = m * X + b
    # Calculate residuals (actual - predicted)
    residuals = Y - Y_pred
    # Return the Sum of Squared Residuals
    return (residuals ** 2).sum()

Key Takeaway: The "best fit" line is mathematically defined as the one that yields the lowest possible Sum of Squared Residuals.

2. Sums of Squares: Baseline and Residual

Regression performance is measured by comparing the total variation in the dependent variable (Y) against the variation remaining after the model is applied. This comparison uses Sums of Squares (SS).

Mechanism: SS Mean (Total Sum of Squares) establishes the baseline error by measuring the squared distance of every Y value from the overall mean of Y ($\bar{Y}$). This represents the variation if the predictor X were ignored.
Execution Flow: SS Fit (Residual Sum of Squares) measures the squared distance of every Y value from the fitted line ($Y_{pred}$). This quantifies the variation that the model failed to explain.
Best Practice: The difference ($ ext{SS Mean} - ext{SS Fit}$) represents the variation explained by the model, which forms the numerator in the R-squared calculation.

import numpy as np

# Assume Y is the dependent variable array and Y_pred is the fitted line
Y_mean = np.mean(Y)

# SS Mean: Variation around the mean (baseline)
SS_Mean = np.sum((Y - Y_mean)**2)

# SS Fit: Variation around the fitted line (residual error)
SS_Fit = np.sum((Y - Y_pred)**2)

Key Takeaway: SS Mean is the total variation to be explained, and SS Fit is the unexplained variation remaining after fitting the line.

3. R-Squared Calculation and Interpretation

R-squared ($R^2$) quantifies the proportion of the total variance in the dependent variable (Y) that is predictable from the independent variable(s) (X). It is the primary measure of model goodness-of-fit.

Mechanism: $R^2$ is calculated as $1 - (\text{SS Fit} / \text{SS Mean})$, which expresses the explained variation as a percentage of the total variation.
Boundary Cases: If the fit is perfect, $\text{SS Fit} = 0$, resulting in $R^2 = 1$ (100% explained); if the fit is no better than the mean, $R^2 \approx 0$.
Nuance: While R-squared is often defined using variance, the raw Sums of Squares (SS) can be used directly because the sample size ($N$) cancels out in the numerator and denominator of the ratio.

# Assuming SS_Mean and SS_Fit have been calculated
SS_Explained = SS_Mean - SS_Fit

R_squared = SS_Explained / SS_Mean

print(f"R-squared: {R_squared:.2f}")
# Example output: R-squared: 0.60 (60% of variance explained)

Key Takeaway: R-squared measures the reduction in error achieved by using the regression line instead of simply using the mean of Y.

4. Multivariate Regression Generalization

The core principles of minimizing squared residuals and calculating R-squared extend directly to models involving multiple predictors (Multiple Linear Regression).

Mechanism: Instead of fitting a 2D line ($Y = mX + b$), multivariate regression fits a hyperplane defined by $Y = m_1X_1 + m_2X_2 + \dots + b$.
Visualization: With two predictors ($X_1, X_2$), the model is visualized as a plane in a 3D space, where the residuals are the vertical distances from the data points to this fitted plane.
Best Practice: The R-squared calculation remains mathematically identical regardless of the number of predictors or the complexity of the model equation, provided the model is fitted by minimizing the Sum of Squared Residuals.

Key Takeaway: The mathematical foundation (Least Squares and SS calculation) is universal for fitting and evaluating any model that minimizes squared error.

Topics Covered in Linear Regression — The Math, Not Just the API

Core Regression Steps (0:20 - 0:58) — Regression analysis involves fitting a line, quantifying performance, and testing statistical significance.
Least Squares Method (1:20 - 2:36) — The best-fit line minimizes the Sum of Squared Residuals (SSR), which are the errors between data and the line.
Model Parameters (2:37 - 3:00) — The fitted line is defined by the estimated slope and Y-intercept, which determine its predictive power.
Baseline Variation (3:15 - 4:14) — SS Mean establishes the total variation in Y that must be explained, calculated around the average Y value.
Residual Variation (4:15 - 4:53) — SS Fit quantifies the unexplained variation remaining after the least squares line has been fitted to the data.
R-Squared Formula (4:54 - 6:32) — R-squared is calculated as the ratio of explained variation to total variation, showing goodness-of-fit.
Boundary Cases (6:33 - 8:18) — R-squared ranges from 0% (no fit) to 100% (perfect fit) and applies universally across model types.
Multiple Predictors (8:19 - 9:13) — Linear regression extends to multiple variables by fitting a plane or hyperplane in higher dimensions.

Statistics for Data Science Cheat Sheet

Residual — Distance from data point to the fitted line
```
residual = Y_actual - Y_predicted
```
Least Squares Method — Minimizes the total Sum of Squared Residuals (SSR)
```
min(sum((Y - (m*X + b))**2))
```
SS Mean — Total variation in Y around the mean of Y
```
np.sum((Y - Y.mean())**2)
```
SS Fit — Unexplained variation remaining around the fitted line
```
np.sum((Y - Y_pred)**2)
```
R-squared — Proportion of Y variance explained by the model
```
R2 = 1 - (SS_Fit / SS_Mean)
```
Multivariate Regression — Linear model using two or more independent variables
```
Y = m1*X1 + m2*X2 + b
```

Comparison Table

SS Metric	Definition	Purpose in R-squared
SS Mean	Total squared distance from Y to $\bar{Y}$.	Denominator (Total Variation)
SS Fit	Total squared distance from Y to $Y_{pred}$.	Numerator (Unexplained Variation)
Variance	Average sum of squares ($ ext{SS}/N$).	Used conceptually, cancels out in ratio.

Common Pitfalls

Mistake: Confusing residual (error) with the model parameters (slope/intercept). Avoid: Residuals are vertical distances; parameters define the line's position.
Mistake: Assuming a high R-squared guarantees a good predictive model. Avoid: R-squared only measures fit; check residual plots for linearity assumptions.
Mistake: Calculating R-squared using raw variance values instead of SS values. Avoid: Use SS values directly since the sample size $N$ cancels out in the ratio.
Mistake: Forgetting to square the residuals before summing them. Avoid: Squaring prevents positive and negative errors from erroneously canceling out.

FAQs

Why do we square the residuals instead of just summing the absolute values? Squaring penalizes larger errors more heavily, ensuring the line is closer to all points. It also makes the error function differentiable, which is necessary for optimization algorithms.
Does R-squared work for non-linear models? Yes, the R-squared calculation based on SS Mean and SS Fit is universal for quantifying the goodness-of-fit of any model equation, provided it minimizes squared error.
What does a non-zero slope imply? A non-zero slope suggests that the independent variable (X) has predictive power over the dependent variable (Y), meaning the fitted line is better than just using the mean.

Common pitfalls

Mistake: Confusing residual (error) with the model parameters (slope/intercept). Avoid: Residuals are vertical distances; parameters define the line's position.
Mistake: Assuming a high R-squared guarantees a good predictive model. Avoid: R-squared only measures fit; check residual plots for linearity assumptions.
Mistake: Calculating R-squared using raw variance values instead of SS values. Avoid: Use SS values directly since the sample size $N$ cancels out in the ratio.
Mistake: Forgetting to square the residuals before summing them. Avoid: Squaring prevents positive and negative errors from erroneously canceling out.

This tutorial provides a practical, step-by-step guide to using **Linear Regression Fundamentals** within *Core Statistical Principles*. After watching, a learner will be able to master the mathematical derivation of model fitting and performance quantification. ### **Curriculum Alignment: Topics Covered** 1. **Core Steps of Linear Regression (00:20 - 00:58)** * **Concept:** Identifying the three primary objectives of regression analysis: model fitting, performance quantification, and statistical significance testing. * **Action:** Outlining the sequential process: Use Least Squares to fit a line $\rightarrow$ Calculate R-squared $\rightarrow$ Calculate a P-value for R-squared. 2. **Least Squares Method (LSM) and Residuals (01:20 - 02:36)** * **Concept:** The method used to determine the best-fit line by minimizing the total error between the data and the line (the residuals). * **Action:** Defining the residual (distance from data to the line); demonstrating the process of rotating the line iteratively until the minimum Sum of Squared Residuals (SSR) is found. 3. **Model Parameters and Prediction Power (02:37 - 03:00)** * **Concept:** Understanding that the fitted line is defined by its Y-intercept and slope, and a non-zero slope suggests predictive power. * **Action:** Extracting the two estimated parameters (slope and intercept) from the least squares fit to form the line equation. 4. **Defining Baseline Variation (Sum of Squares around the Mean) (03:15 - 04:14)** * **Concept:** Establishing a baseline measure of total variation (SS Mean) in the dependent variable (Y) when ignoring the predictor (X). * **Action:** Calculating SS Mean by summing the squared distances of each data point to the average value of Y; relating SS Mean to the calculation of raw variance. 5. **Defining Model Residual Variation (Sum of Squares around the Fit) (04:15 - 04:53)** * **Concept:** Quantifying the unexplained variation that remains *after* the least squares line is fitted (SS Fit). * **Action:** Calculating SS Fit by summing the squared distances of each data point to the fitted regression line. 6. **R-Squared Calculation and Interpretation (04:54 - 06:32)** * **Concept:** Determining the goodness-of-fit, or the percentage of the variation in the dependent variable that is explained by the independent variable. * **Action:** Applying the R-squared formula using variations or sums of squares (Variation Mean - Variation Fit) / Variation Mean and interpreting the result as the reduction in variance (e.g., 60% explained). 7. **Boundary Cases and Generalized Application of R-Squared (06:33 - 08:18)** * **Concept:** Demonstrating the limiting values of R-squared (100% for perfect fit, 0% for no predictive power) and establishing that the R-squared calculation is universal for any model equation. * **Action:** Analyzing data scenarios where the variation around the fit is zero or equal to the variation around the mean; applying the SS calculation method to an arbitrary complex equation. 8. **Extension to Multiple Linear Regression (Fitting a Plane) (08:19 - 09:13)** * **Concept:** Generalizing the linear regression principle to models involving multiple predictors (Multivariate Regression). * **Action:** Visualizing the representation of a model with two predictors (weight and tail length) which requires fitting a plane in a 3D space rather than a 2D line. ### **Learning Outcomes** * **Proficiency:** Ability to define and mathematically calculate the core statistical components of linear regression, including residuals, Sum of Squares Mean (SS Mean), and R-squared. * **Workflow Management:** Ability to follow the end-to-end conceptual process of fitting a model using least squares and subsequently quantifying its explanatory power. * **Tooling:** Comfort with statistical terminology related to variation and explanation, differentiating between variance (average sum of squares) and raw sums of squares. ### **Advanced/Tricky Concepts** * **Variance vs. Sum of Squares:** While the R-squared ratio works using either variation (variance) or raw sums of squares (SS), the distinction is critical. Variance is the average sum of squares ($\text{SS} / N$), and using the raw $\text{SS}$ implies that the sample size ($N$) cancels out in the numerator and denominator, simplifying the calculation but potentially obscuring the underlying definition of variance.

Frequently asked questions

Why do we square the residuals instead of just summing the absolute values?

Squaring penalizes larger errors more heavily, ensuring the line is closer to all points. It also makes the error function differentiable, which is necessary for optimization algorithms.

Does R-squared work for non-linear models?

Yes, the R-squared calculation based on SS Mean and SS Fit is universal for quantifying the goodness-of-fit of any model equation, provided it minimizes squared error.

What does a non-zero slope imply?

A non-zero slope suggests that the independent variable (X) has predictive power over the dependent variable (Y), meaning the fitted line is better than just using the mean.

How was this lesson?

Your feedback helps us refine explanations and catch bugs.