• In Machine Learning, to prevent overfitting in regression models, we can reduce the impact of each feature by using regularization techniques.

    • Ridge Regression is a type of L2 regularization that adjusts the model for linear models that are not responsive.
      • This is done by manipulating the Alpha value.
        • A higher Alpha value imposes more constraints on the coefficients, leading to generalization.
        • As Alpha approaches 0, the regularization effect weakens.
        • The Learning Curve shows that if there is a lot of data, regularization is not very relevant.
    • Lasso Regression (L1 regularization) can result in some coefficients becoming exactly 0, making it a more extreme form of regularization.
  • Choosing between Ridge and Lasso:

    • Start with Ridge.
    • If it is expected that most of the features are not meaningful, try Lasso.
    • Also try Lasso when a simpler model is desired.
  • There is also ElasticNet, which combines the benefits of both Ridge and Lasso.

    • It is practically useful, but requires tuning two parameters.
  • In classification problems, the effect is as shown in the graph below:

    • The degree of regularization is determined by C.
      • A higher C value weakens the regularization (opposite of Alpha).
      • In other words, weaker regularization leads to overfitting, making the model more sensitive to small details (as shown in the third graph). image #Getting Started with Machine Learning in Python