Hyperparameter tuning for Ensemble of ML models (Simple Python Example)

If you have created an ensemble of ML models in scikit-learn, and you want to improve its performance even further, you can tune this ensemble itself.It can be easily implemented by GridSearchCV and VotingClassifier from Scikit-Learn. The main idea is to set the hyperparameters you want to search the best combination of values on, then set all possible weights of Machine Learning classifiers you have in your situation. For example, one option could be that your Random Forest classifier is more important that Linear Regression classifier, and opposite.Then the search for the best combination of your selected hyperparameters will do almost all the job.

That means that by tuning ensembles of ML models you can get the same accuracy as you got from cross validation score in ensemble learning mannier, or increase it with the tuning procedure. So, the good idea is to always invest some time tuning your ensemble. 

In this example we used following ML models for binary classification problem:
- Logistic Regression classifier: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html
- Random Forest classifier: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
- Decision Tree classifier: https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html

Also we used following methods originally from scikit-learn:
- GridSearchCV: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html
- Voting Classifier: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html
- Cross Validation Score: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

By using all of these functionalities we implemented the search across Soft and Hard voters in Grid Search CV. The main differences and examples of calculations are presented in the BONUS part of this video.

The content of this video:
0:00 - Creating Ensemble of  Machine Learning models1:00 - Tuning the Ensemble of ML models
9:58 - Results of tuning (accuracy).
10:56 - BONUS part: Soft Voting and Hard Voting

#hyperparametertuning #ensemblelearning #gridsearchcv

Enjoy!