FeatureFlex: An AutoML Project with Advanced Feature Selection

Overview

FeatureFlex is an AutoML project that provides a comprehensive suite of machine learning capabilities. It includes advanced preprocessing, feature selection, model optimization, and evaluation functionalities, making it a robust choice for tackling classification tasks with large and complex datasets.

This package is particularly suited for tasks requiring feature selection and comparison across multiple methods, including SHAP, Boruta, SelectKBest, and ReliefF.

---

Features

• Advanced Feature Selection:

  • Boruta (GitHub - BorutaPy)
  • SelectKBest (Scikit-learn Documentation)
  • SHAP-based feature selection
  • ReliefF (via scikit-rebate: GitHub - scikit-rebate)

• Dynamic Model Optimization:

  • Grid search
  • Random search
  • Bayesian optimization

• Evaluation Metrics:

  • AUC, Accuracy, Precision, Recall, F1-score
  • Confusion Matrix
  • ROC and Precision-Recall Curves

• Comparison Tool:

  • Compare feature selection methods based on model performance.

---

Installation

Install FeatureFlex via PyPI:

pip install FeatureFlex

Additional Dependencies for Feature Selection

To use ReliefF, install scikit-rebate:

pip install skrebate

To use Boruta, install BorutaPy:

pip install boruta

---

Usage

Preprocessing Data

FeatureFlex includes a DataPreprocessor for preprocessing data with missing values, scaling, and encoding.

from preprocessing import DataPreprocessor

data = ...  # Load your dataset
preprocessor = DataPreprocessor()
X, y, _ = preprocessor.preprocess(data, target_column="click")

Feature Selection

FeatureFlex allows you to use various feature selection techniques:

Using SHAP-based Feature Selection

from feature_selector import EnhancedFeatureSelector

selector = EnhancedFeatureSelector(input_dim=X.shape[1])
top_features = selector.select_via_shap(X, y, n_features=10)

Using Boruta

from boruta import BorutaPy
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(random_state=42)
boruta_selector = BorutaPy(rf, n_estimators='auto', random_state=42)
boruta_selector.fit(X, y)
selected_features = X[:, boruta_selector.support_]

Using SelectKBest

from sklearn.feature_selection import SelectKBest, f_classif

selector = SelectKBest(score_func=f_classif, k=10)
selected_features = selector.fit_transform(X, y)

Using ReliefF

from skrebate import ReliefF

selector = ReliefF(n_features_to_select=10)
selector.fit(X, y)
selected_features = selector.transform(X)

Model Optimization

Optimize models using dynamic, grid, random, or Bayesian search:

from model_optimizer import ModelOptimizer

optimizer = ModelOptimizer()
best_model, best_score = optimizer.optimize_model(X, y, method="bayesian")

Comparison of Feature Selection Methods

Compare different feature selection methods using the provided comparison script:

from comparison import compare_feature_selectors

results = compare_feature_selectors(data, target_column="click", n_features=10)

Evaluation

Evaluate your model with various metrics and generate reports:

from evaluation import ModelEvaluator

metrics = ModelEvaluator.evaluate(model, X_test, y_test, output_format="html")

---

Example Dataset

The package includes utilities tested with datasets such as:

• Avazu CTR Prediction Dataset (50k random rows)

For more information on the dataset's context, see:

• AutoML Research

---

Comparison with Existing Packages

FeatureFlex distinguishes itself from existing feature selection packages:

Method	FeatureFlex	Boruta	SelectKBest	SHAP	ReliefF
Multi-method support	✔	✘	✘	✘	✘
Dynamic optimization	✔	✘	✘	✘	✘
Scalable to large datasets	✔	✔	✔	✔	✔
Integrated evaluation	✔	✘	✘	✘	✘

---

Full Comparison Script

Here is an example script to compare various feature selection techniques:

from comparison import compare_feature_selectors, save_results_and_plots
import pandas as pd

# Load dataset
data = pd.read_csv("path/to/dataset.csv")
results = compare_feature_selectors(data, target_column="click", n_features=10)

# Save results to CSV and generate plots
save_results_and_plots(results)

---

Dependencies

FeatureFlex depends on the following libraries:

• Core:

  • numpy, pandas, scikit-learn
  • matplotlib, shap, optuna

• Feature Selection:

  • BorutaPy (GitHub)
  • scikit-rebate (GitHub)

• Optimization:

  • optuna

Refer to the requirements.txt file for the full list of dependencies.

---

Project Links

• GitHub: FeatureFlex Repository
• PyPI: FeatureFlex on PyPI

---

License

This project is licensed under the MIT License.

---

Contact

For queries or suggestions, contact the author at [email protected].

---

Changelog

FeatureFlex uses a dynamic versioning system. The current version is updated automatically at build time.

---

Contribution

Contributions are welcome! Feel free to fork the repository and submit a pull request.