Feature Engineering and Addressing Overfitting Using Elastic Net Regularization
In this project, we performed feature engineering to preprocess and organize the data into a readable and structured format suitable for analysis. We visualized the data using plots such as box plots and scatter plots to understand the relationships between variables and identify potential outliers or patterns.
After preparing the data, we trained our initial model. However, the model exhibited overfitting, which indicated that it was performing well on the training data but not generalizing effectively to unseen data. To address this issue, we implemented regularization techniques, including Lasso Regression, Ridge Regression, and Elastic Net.
Among these methods, Elastic Net proved to be the most effective in reducing overfitting. By combining the strengths of both Lasso and Ridge, Elastic Net helped us strike a balance between feature selection and model complexity, ultimately improving the model's performance and ensuring better generalization to new data.