Ridge Regression: Stable Coefficients When Predictors Are Correlated

Your regression model has ten predictors and the coefficients are wild. One is positive 500, another is negative 480, and they flip sign every time you add or remove a variable. The predictions might even look decent, but the coefficients are hard to trust. This is what multicollinearity does to ordinary least squares. Ridge regression adds a penalty that shrinks coefficients toward zero, which makes the estimates less sensitive to the particular sample.

Worked Example (Simulated): OLS and Ridge When Two Inputs Move Together

Question: How do ordinary least squares (OLS) and ridge fit an outcome when two inputs move together?

Inputs: Simulated data, seed 17. The script makes 120 rows with three inputs. The data are built with true weights of 4 for x1, 4 for x2 and 1 for x3, plus random noise. Inputs x1 and x2 are built from one shared signal plus a small amount of noise (standard deviation 0.08), so they are near-copies of each other. The script scales each input to mean zero and standard deviation one.

Method: The script fits OLS and ridge. Ridge shrinks estimates toward zero. Five-fold cross-validation tests penalty sizes on held-out rows and picks ridge's alpha, or penalty size.

import numpy as np
from sklearn.linear_model import LinearRegression, RidgeCV
from sklearn.preprocessing import StandardScaler

# Simulated data, seed 17. The shared signal makes x1 and x2 correlated.
rng = np.random.default_rng(17)
n = 120
shared = rng.normal(size=n)
X_raw = np.column_stack([
    shared + rng.normal(scale=0.08, size=n),
    shared + rng.normal(scale=0.08, size=n),
    rng.normal(size=n),
])
y = 4 * X_raw[:, 0] + 4 * X_raw[:, 1] + X_raw[:, 2] + rng.normal(scale=1.5, size=n)

# Scale each predictor to mean 0 and standard deviation 1.
scaler = StandardScaler()
X = scaler.fit_transform(X_raw)
ols = LinearRegression().fit(X, y)
alphas = np.logspace(-3, 3, 61)
ridge = RidgeCV(alphas=alphas, cv=5, scoring="neg_mean_squared_error").fit(X, y)

print("Data: simulated, seed 17")
print("Rows:", n)
print("CV folds: 5")
print("Predictors: x1, x2, x3; x1 and x2 share a signal")
print("OLS coefficients:", np.round(ols.coef_, 3).tolist())
print("Ridge chosen alpha:", ridge.alpha_)
print("Ridge coefficients:", np.round(ridge.coef_, 3).tolist())
print("OLS training RMSE:", round(np.sqrt(np.mean((y - ols.predict(X)) ** 2)), 3))
print("Ridge training RMSE:", round(np.sqrt(np.mean((y - ridge.predict(X)) ** 2)), 3))

Printed output:

Data: simulated, seed 17
Rows: 120
CV folds: 5
Predictors: x1, x2, x3; x1 and x2 share a signal
OLS coefficients: [2.87, 5.871, 1.11]
Ridge chosen alpha: 0.3981071705534973
Ridge coefficients: [3.494, 5.232, 1.103]
OLS training RMSE: 1.377
Ridge training RMSE: 1.379

Result: The OLS estimates for x1 and x2 are 2.87 and 5.871, while ridge gives 3.494 and 5.232. The built-in weights are the same for both inputs, and ridge's two estimates sit closer to each other than OLS's two. The training errors are close: 1.377 for OLS and 1.379 for ridge.

Limit: The example does not test error on new data, so it does not show that ridge predicts better. The estimates are from one simulated sample, and they do not show which input caused the outcome.

Sources

• Hoerl A, Kennard R (1970). Ridge regression: biased estimation for nonorthogonal problems. Technometrics. doi 10.1080/00401706.1970.10488634
• Friedman J, Hastie T, Tibshirani R (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning (2nd ed.). statlearning.com

What Is Ridge Regression?

Ridge regression is linear regression with a guardrail. It fits the same kind of model (a weighted combination of predictors to predict an outcome) but it adds a penalty term that punishes large coefficients. This penalty is called L2 regularization, and it works by adding the sum of squared coefficients (multiplied by a tuning parameter called lambda) to the loss function that the model minimizes.

For a standard linear regression, try the free linear regression tool.

The practical effect is that coefficients get pulled toward zero. They never reach exactly zero (that is what lasso does), but they shrink enough to become stable. When you have correlated predictors: like TikTok ad spend, Facebook ad spend, and Google ad spend that all tend to increase together during holiday campaigns: ordinary least squares (OLS) cannot figure out which channel deserves credit. It might assign a huge positive coefficient to one and a huge negative coefficient to another, and those estimates are essentially random. Ridge shrinks estimates for inputs that move together. Their signs and sizes depend on the data, input scale, and penalty. These estimates do not show what caused the outcome.

The key concept is the bias-variance tradeoff. If the straight-line model is right and errors average to zero for each set of inputs, OLS estimates are right on average across many samples. One estimate can still be far from the true value. But when predictors are correlated, the variance of those estimates is enormous, meaning any single dataset gives you wildly different answers. Ridge introduces some bias (coefficients are systematically pulled toward zero) in exchange for lower variance. Ridge shrinks estimates. This can lower errors on new data. The result depends on the data and penalty.

Consider a marketing mix model with five advertising channels. The channels are correlated because budgets tend to move together: when you increase total marketing spend, you increase most channels simultaneously. The worked example above shows OLS and ridge estimates from made-up data with inputs that move together. It does not show cause and effect. Ridge gives adjusted estimates for the channels. Their signs and sizes depend on the data, input scale, and penalty. They do not show each channel's own effect.

When to Use Ridge Regression

The clearest signal that you need ridge regression is when your OLS coefficients are unstable or implausibly large. If you run a regression and some coefficients are in the hundreds or thousands (or they change dramatically when you add or remove a single observation) multicollinearity is the likely culprit. The Variance Inflation Factor (VIF) shows how much one predictor overlaps with the others. Ridge is one way to handle overlapping predictors; whether to act depends on the data and the goal.

Ridge is also the right choice when you have many predictors relative to your sample size. With 50 predictors and 100 observations, OLS may overfit and may predict new data poorly. It need not fit the training data perfectly. Ridge regularization constrains the model, preventing it from over-committing to noise in the training set. In the extreme case where you have more predictors than observations, OLS cannot even produce a unique solution. Ridge handles this gracefully because the penalty term makes the problem solvable regardless of the predictor-to-sample ratio.

In genomics research, you might have gene expression measurements for 20,000 genes but only 200 patients. In finance, you might model stock returns using dozens of correlated economic indicators (GDP growth, unemployment rate, consumer confidence, interest rates) that all move together during economic cycles. In e-commerce, you might predict customer lifetime value from 30 behavioral features (pages visited, time on site, email opens, cart additions) that overlap heavily. All of these are classic ridge regression problems.

A crucial distinction: ridge regression keeps all predictors in the model. It shrinks coefficients toward zero but never sets them to exactly zero. If you believe that only a handful of your predictors actually matter and the rest are irrelevant, lasso regression is a better choice because it performs variable selection by driving some coefficients to exactly zero. If you want to keep all predictors, because they all have theoretical justification, or because you care more about prediction accuracy than identifying which variables matter most, ridge is what you want.

What Data Do You Need?

You need a CSV with one numeric outcome column and one or more numeric predictor columns. The tool requires you to map at least one outcome and one predictor. You can include as many additional predictors as you need, the tool accepts predictor_[N] mappings for any number of features.

All columns must be numeric. If you have categorical predictors (like region or product category), you will need to convert them to dummy variables before uploading. The outcome should be continuous: revenue, score, weight, duration, concentration. For binary outcomes (yes/no, pass/fail), use logistic regression instead.

Ridge regression is especially useful when your predictors are correlated. If they are not correlated, ridge will still work. It will just produce results very similar to OLS, because there is no instability to fix. You can check for multicollinearity by looking at the correlation matrix of your predictors before running the analysis, but in practice, just run both OLS and ridge and compare. If the coefficients are similar, you did not need ridge. If they are dramatically different, ridge is doing important work.

Ridge handles the case where predictors outnumber observations. The tool uses cross-validation to select the optimal lambda, and with very small samples, cross-validation becomes unreliable, so interpret the results cautiously.

How to Read the Report

The report contains nine sections, starting with an overview and executive summary that highlight the key findings in plain language. Here is what to focus on in the technical sections.

The Cross-Validation Error plot is where lambda selection happens. It shows the prediction error (typically mean squared error) on the y-axis and log(lambda) on the x-axis. You will see a U-shaped curve: at very small lambda values, the model is essentially OLS and may overfit. At very large lambda values, all coefficients are crushed toward zero and the model underfits. The optimal lambda sits at the bottom of the U: the sweet spot where the model is regularized enough to generalize but not so much that it loses signal. The report marks this optimal lambda with a vertical line. A second line often marks the "one standard error" lambda: a slightly more regularized model that is within one standard error of the minimum, which some practitioners prefer for extra stability.

The Ridge Trace (coefficient path plot) shows how each predictor's coefficient changes as lambda increases from left to right. At the left edge (low lambda), coefficients are close to their OLS values. As lambda grows, they shrink toward zero. Predictors with strong, robust signal maintain larger coefficients across a wide range of lambda values. Predictors that are mostly noise drop to near-zero quickly. This plot is one of the most informative outputs. It shows which predictors the model relies on most and which ones are fragile. If two predictors have coefficients that move in mirror image (one goes up as the other goes down), they are highly collinear and ridge is splitting the effect between them.

The Coefficient Comparison section shows the final ridge coefficients alongside the OLS coefficients. This is where you see the stabilization effect most clearly. Large, volatile OLS coefficients become moderate ridge coefficients. Look for predictors where the sign changed between OLS and ridge. If a sign changes, check whether inputs move together. See how estimates change with the data and penalty. Ridge can help with prediction, but its estimates do not show which inputs matter or cause the outcome.

The Actual vs Predicted plot and Model Performance metrics (R-squared, RMSE, MAE) tell you how well the model fits. Compare the ridge R-squared to the OLS R-squared. Ridge will typically have a slightly lower training R-squared (because the penalty prevents perfect fitting), but it often has a higher R-squared under cross-validation (because it generalizes better). If the training R-squared is much higher than the R-squared under cross-validation, OLS was overfitting and ridge is protecting you from it.

When to Use Something Else

If you want to identify which predictors actually matter, and discard the rest, use lasso regression instead. Lasso uses L1 regularization, which drives some coefficients to exactly zero. This is effectively automatic variable selection. Ridge keeps everything; lasso makes hard choices. If you are building a model for interpretation ("which three factors drive churn?") rather than pure prediction, lasso gives you a cleaner answer.

If you want a compromise between ridge and lasso, elastic net combines both penalties. It tends to select groups of correlated predictors together (like ridge) while still dropping irrelevant ones (like lasso). Elastic net is an option when you are unsure whether you need variable selection.

If your predictors are not correlated and you have far more observations than predictors, ordinary least squares is simpler and gives unbiased estimates. Ridge will not hurt in this case, it will just produce nearly identical results, but there is no benefit to the added complexity.

If the multicollinearity is so severe that you want to reduce your predictors to a smaller set of uncorrelated components, consider PCA (principal component analysis) followed by regression on the components. This approach is called PCR (principal component regression). It solves collinearity by transforming the predictors into orthogonal dimensions. The tradeoff is that the components are harder to interpret than the original predictors, "PC1 explains 40% of variance" is less actionable than "email spend has a coefficient of 3.2."

For non-linear relationships, tree-based methods like random forest or XGBoost do not suffer from multicollinearity at all. They split on one predictor at a time and handle correlated features naturally. If prediction accuracy is all you care about and you do not need coefficient interpretation, these methods often outperform ridge regression.

The R Code Behind the Analysis

Every report names its method and includes the R code that produced it, so you can reproduce, audit and cite it. The method code is fixed, not rewritten for each run, so the same data gives the same analysis every time.

The analysis uses the glmnet package with alpha = 0 (which specifies ridge regression, alpha = 1 would be lasso, and values between 0 and 1 give elastic net). Lambda selection is done via cv.glmnet(), which performs k-fold cross-validation across a grid of lambda values and selects the one that minimizes prediction error. The coefficient path plot is generated from the full glmnet() fit, showing how each coefficient evolves across the entire regularization path. The report uses glmnet (Friedman, Hastie and Tibshirani, 2010) and cv.glmnet(). Every step is visible in the code tab of your report, so you or a statistician can check exactly what was done.