Home / Supervised ML / Model Diagnostics

Supervised ML · Chapter 5 · 25 min read

Model Diagnostics: Is Your Regression Actually Valid?

You've run your OLS regression, got your coefficients, and your $R^2$ looks decent. But does your model actually satisfy the assumptions it relies on? If the errors aren't normally distributed, your p-values are wrong. If the variance of the errors changes with $X$, your standard errors are wrong. And if you keep adding variables to chase a higher $R^2$, you're building a model that only works on the data you already have.

This page teaches you how to diagnose those problems — and how to compare models honestly.

Part 1: Checking Model Assumptions

All OLS inference (t-tests, F-tests, confidence intervals) assumes that the model's errors follow certain patterns. We can't observe the true errors $u_i$, but we can look at the OLS residuals $\hat{u}_i$ as stand-ins. The key idea: if the model is correct, the residuals should behave like a well-behaved normal distribution centred at zero with constant spread.

Assumption 1: Normally Distributed Errors

If normality holds, 95% of your residuals should fall within $\pm 2\hat{\sigma}$ of zero. Any residual larger than $|2\hat{\sigma}|$ is a potential outlier — and too many of those signal a problem.

Visual Check 1: Histogram of Residuals

The fastest way to get a first impression. Plot a histogram of $\hat{u}_i$. If normality holds, it should look bell-shaped — one peak in the middle, tapering off symmetrically on both sides, with no heavy tails or extreme outliers dragging one side out.

Two things to watch for: skewness (the bump is shifted left or right, with a long tail on one side) and bimodality (two bumps, which suggests two different data-generating processes got mixed together).

Visual Check 2: QQ-Plot (Quantile-Quantile Plot)

A histogram is subjective — it depends on the bin width. A QQ-plot is more precise. It plots the actual quantiles of your residuals on the y-axis against the theoretical quantiles of a perfect normal distribution on the x-axis.

The theoretical normal line always has a 45° slope, because a perfect normal distribution would mean: actual quantile = theoretical quantile at every point. Deviations from this line reveal the type of non-normality you're dealing with.

Formalising It: Skewness and Kurtosis

Histograms and QQ-plots are visual — they give you a feel, not a number. The skewness coefficient $m_3$ and kurtosis coefficient $m_4$ turn those visual impressions into statistics you can compute and compare.

Skewness Coefficient $m_3$

Skewness measures whether one tail of the residual distribution is heavier than the other. A perfectly symmetric distribution has $m_3 = 0$.

Interpreting the sign: a positive $m_3$ means the distribution has a long right tail (more extreme positive residuals). A negative $m_3$ means a long left tail. A value of zero means perfect symmetry.

Kurtosis Coefficient $m_4$

Kurtosis measures the "peakedness" of a distribution — how much probability is concentrated in the tails versus the centre. The normal distribution has $m_4 = 3$. Think of that as the reference.

If $m_4 > 3$: the distribution has heavier tails than normal (leptokurtic — more extreme outliers than expected). If $m_4 < 3$: lighter tails, fewer extremes (platykurtic). For your OLS residuals to be consistent with normality, $m_4$ should be close to 3.

The Jarque-Bera Test: A Formal Normality Test

Instead of eyeballing a histogram, the Jarque-Bera test gives you an actual hypothesis test for normality.

The null hypothesis is: the errors follow a normal distribution. The idea is that under normality, both $m_3$ should be close to 0 and $m_4$ should be close to 3. The Jarque-Bera statistic combines both into a single number:

The critical value of $\chi^2(2)$ at the 5% level is 5.991. If $J > 5.991$, you reject $H_0$ — meaning the residuals are significantly non-normal.

Assumption 2: Homoskedasticity

OLS assumes the variance of the errors is constant across all observations: $\text{Var}(u_i | X_i) = \sigma^2$ for every $i$. This is called homoskedasticity.

The opposite — where variance changes with $X$ or with the size of $Y$ — is called heteroskedasticity. It doesn't bias your coefficient estimates, but it makes your standard errors wrong, which means your t-statistics and p-values are unreliable.

Heteroskedasticity is common whenever you're dealing with data that spans a large range: wealthier households have more variable spending than poorer ones; larger firms have more variable profits; cities with bigger populations have more variable crime rates.

The quickest diagnostic: plot the residuals $\hat{u}_i$ against the fitted values $\hat{Y}_i$ (or against $X$). If homoskedasticity holds, the spread of the dots should look roughly the same across the whole range — a horizontal band of constant width. A funnel shape (spreading out to the right) is the classic sign of heteroskedasticity.

Assumption 3: Zero-Mean Errors — $E(u\,|\,X) = 0$

This assumption says: on average, the model's errors are zero, regardless of the value of $X$. If this is violated, your model has a specification error — it's systematically wrong in a predictable way.

What does that mean concretely? If $E(u|X) > 0$, the true value of $Y$ will always be higher than what your model predicts — you're consistently underestimating. If $E(u|X) < 0$, you're consistently overestimating.

This assumption is typically violated in two situations. First, you've left out an important predictor that belongs in the model (omitted variable bias). Second, you've used the wrong functional form — for example, the true relationship is quadratic but you've modelled it as linear, so your residuals will systematically curve upward then downward across the range of $X$.

To check: plot the residuals and draw a horizontal line at their mean. If the mean line runs close to zero across the full range of $X$, the assumption is roughly satisfied. If the residuals have a curve or a trend pattern — clustering positive on one side and negative on the other — you have a specification problem.

Part 2: Choosing the Best Model

Once your model passes the diagnostic checks, a new question arises: is it the right model, or would adding (or removing) a variable give you something better?

Why $R^2$ Alone Will Mislead You: Overfitting

Here's an uncomfortable fact: $R^2$ can only increase (or stay the same) when you add a variable to a regression. It is mathematically impossible for $R^2$ to decrease, no matter how useless the new variable is — at worst, its coefficient is estimated as exactly zero, contributing nothing.

This creates the overfitting trap. If you keep adding variables to maximise $R^2$, you'll end up with a model that fits your specific dataset perfectly but fails badly on any new data. The model has memorised the noise in your sample rather than the underlying pattern.

$R^2$ is useful only at the extremes: when it's close to 1, the model explains nearly all variation; when it's close to 0, the predictors are barely better than nothing. Anywhere in between, you need better tools.

Adjusted $R^2$: Penalising Complexity

The adjusted $R^2$ (written $\bar{R}^2$) fixes the overfitting problem by penalising each additional variable you add. Unlike plain $R^2$, it can actually decrease when you add a variable that doesn't explain enough new variation to justify its presence.

The logic: adding variable $K+1$ increases $R^2$, but it also increases $K$, shrinking $(n-K-1)$ and making the penalty term larger. $\bar{R}^2$ rises only if the new variable increases $R^2$ enough to overcome that growing penalty. If the variable is useless, $\bar{R}^2$ actually falls. Always aim to maximise $\bar{R}^2$, not $R^2$.

Information Criteria: AIC and BIC

Information criteria provide another way to compare models, and they work across a wider range of situations than adjusted $R^2$. The two most common are the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Both penalise model complexity but in slightly different ways.

The rule is simple: lower is better. When comparing two models, pick the one with the smaller AIC (or BIC). The values can be negative — that's fine. A more negative value is still better; it means the model fits well without needing many parameters.

AIC and BIC often agree, but when they conflict, AIC tends to favour slightly more complex models while BIC favours simpler ones (because its penalty grows with $\ln(n)$, which is larger than 2 for $n \geq 8$).

Comparing a Linear Model to a Log-Linear Model

One tricky situation: AIC and BIC cannot be directly compared between a linear model and a log-linear model. The reason is that the dependent variable is different — in one it's $Y$, in the other it's $\ln(Y)$ — so the residuals are on different scales and their RSS values are not comparable.

To fix this, we apply a correction to the log-linear model's AIC or BIC before comparing:

After correction, compare normally: lower corrected AIC wins. (The same correction applies to BIC.)

Worked Example: Diagnosing a Salary Regression