Regularization (Ridge, Lasso and ElasticNet)

 Regularization:

Over fitting is caused because of over optimization of cost function in training dataset.

Ridge Regression (L2 Regularization):-

1.      Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated.

2.      By adding a degree of bias to the regression estimates, ridge regression reduces the standard errors.

Cost function for Ridge Algorithms:

Lasso Regression (L1 Regularization):-

1. Lasso Regression (L1) is similar to ridge, but it also performs feature selection.

2. It will set the coefficient value for features that do not help in decision making very low, potentially zero.

ElasticNet Regression: 

ElasticNet linear regression (L1 + L2 Regularization):

In elastic Net Regularization we added the both terms of L1 and L2 to get the final loss function. Elastic Net combines the strength of ridge and lasso regression both.

When to use: - If you are not sure about when to use Ridge OR Lasso then use ElasticNet.

Cost function for ElasticNet Algorithms:

Elastic Net combines the strength of ridge and lasso regression both.

Why ElasticNet Regression:

Elastic net linear regression uses the penalties from both the lasso and ridge techniques to regularize regression models. The technique combines both the lasso and ridge regression methods by learning from their shortcomings to improve the regularization of statistical models.

Cost function for ElasticNet Algorithms:

We can clearly see here: 

elastic_net_penalty = (alpha * l1_penalty) + ((1 – alpha) * l2_penalty) 

 Steps to execute ElasticNet Regularization

from sklearn.linear_model import ElasticNet 

from sklearn.metrics import r2_score 

parameters = {'alpha': 0.1, 'l1_ratio': 0.1 } 

model = ElasticNet(**parameters) 

model.fit(X, y)

y_pred = model.predict(X) 

score = r2_score(y, y_pred) 

print("R2 {}".format(score)) #OUT:

By - Uttam Kumar