regularization - ML

How does regularization affect the optimization process?

Regularization allows the loss function to be minimized effectively, often enabling optimization via stochastic gradient descent.

What is the purpose of setting aside samples for testing in model evaluation?

Setting aside samples for testing allows for an unbiased evaluation of the model's generalization performance.

What is the formula for calculating the weights with L2 regularization?

The weights can be calculated using the formula w = (λI + X^TX)⁻¹X^Ty.

What is the significance of tracking training and validation loss?

Tracking training and validation loss helps identify when a model begins to overfit, allowing for timely intervention.

What is the model error on the test set with regularization compared to without?

With regularization, the model error on the test set is 1.8, while without regularization, it is 2.2.

What is overfitting in machine learning?

Overfitting occurs when a highly parameterized model fits the training data too well, leading to worse performance on new, unseen data.

What does a polynomial model of degree 11 illustrate in the context of overfitting?

A polynomial model of degree 11 can fit the training data closely, but it may not generalize well to new data due to overfitting.

What is the effect of adding L2 regularization to the loss function?

Adding L2 regularization to the loss function modifies it to J(w) = 1/2 Σ(h(xi; w) - yi)² + 1/2 λ||w||².

What is the impact of noise in the data on model performance?

Noise in the data can lead to higher complexity in the model, which may result in overfitting and poor performance on new samples.

How can overfitting be detected during model training?

Overfitting can be detected by tracking the training loss versus the validation loss; when the validation loss diverges while training loss continues to decrease, overfitting is occurring.

What does a decrease in the mean of the absolute values of weights indicate?

A decrease in the mean of the absolute values of weights indicates that regularization has successfully constrained the model's complexity.

What is the consequence of a model having too many parameters?

A model with too many parameters may fit noise in the training data, leading to poor generalization on unseen data.

What does the term 'generalization' refer to in machine learning?

Generalization refers to the model's ability to perform well on unseen data, not just the training data.

What is the relationship between model complexity and generalization?

Higher model complexity can lead to overfitting, where the model fails to generalize well to unseen data.

How can the weights of a model indicate overfitting?

If the weights of a highly parameterized model take on large values, it may indicate overfitting.

How is L2 regularization mathematically expressed?

L2 regularization is expressed as E(w) = 1/2 ||w||² = 1/2 w^Tw.

What happens to the mean absolute weights of a model when L2 regularization is applied?

The mean absolute weights decrease significantly, indicating that regularization has effectively reduced the complexity of the model.

What is the most common form of regularization?

The most common form of regularization is penalizing the weights from taking high values.

What is the role of the optimizer in regularization?

The optimizer minimizes the modified loss function that includes the regularization term.

What is the primary solution to overfitting?

The primary solution to overfitting is regularization.

What is the purpose of the regularization hyperparameter λ?

The regularization hyperparameter λ controls the strength of the regularization applied to the model.