For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind?

Question

For a square matrix of random data, N columns and N rows. I am fitting two models, linear regression and Lasso.
For the linear regression, I achieve a perfect score in train set, while in the Lasso I achieve a score of 0.
import pandas as pd
import numpy as np
from sklearn import linear_model

N = 100
X = pd.DataFrame(np.random.rand(N,N))
y = np.random.randint(20, size=N)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Linear regression score 1.0
Lasso score 0.0
My questions will be:

Can someone give me an intuitive idea why for a square matrix, Lasso regression achieve different results from Linear regression?

What is the intuition behind the Lasso shrinking?

Note: In this case, the matrix is squared and the data/target is created randomly.

Peter · Answer

A few things going on here:

Your matrix is 100x100. So you have no degrees of freedom left in a linear model, which will cause $R^2=1$. See this post.
You use random numbers. Thus, they should make little sense in terms of explaining your dependent variable (it's basically noise). Since Lasso "shrinks" parameters which are not useful (and none is useful given perfect noise), all parameters are set to zero. This in turn also gives you an $R^2=0$. In this case the model consists only of an intercept (should be equal to 0.5), which is essentially the mean.

Change the matrix to have less columns than rows and you will get a low (but positive) $R^2$ for linear regression. Also look at the Lasso coefficients, they are all zero.

Have a look at the predicted values as well. You will see that the linear model does not work well. So the R-squared gives no indication about model fit in this case.

You can also try to change the random pattern to have some trend and you will find that Lasso does not shrink all coefficients to zero.

import pandas as pd
import numpy as np
from sklearn import linear_model

n = 1000
k = 5

X = pd.DataFrame(np.random.rand(n,k))
y = np.random.randint(2, size=n)

lin = linear_model.LinearRegression().fit(X,y)
lasso = linear_model.Lasso().fit(X,y)

print(lin.coef_)
print(lin.intercept_)

print(lasso.coef_)
print(lasso.intercept_)

print('Linear regression score',lin.score(X,y))
print('Lasso score',lasso.score(X,y))

Output:

[-0.00446436 -0.04509905  0.07870413 -0.00165025 -0.03951546]
0.5010650787133537
[-0. -0.  0. -0. -0.]
0.495
Linear regression score 0.003094912043144493
Lasso score 0.0

For a square matrix of data, I achieve $R^2=1$ for Linear Regression and $R^2=0$ for Lasso. What's the intuition behind?

One Answer

Add your own answers!

Ask a Question