MODEL VALIDATION-Hyperparameter
MODEL VALIDATION-Hyperparameter
Model validation refers to the process of confirming that the model achieves its intended purpose , how effective our model is.
1- Hold Out Method
Hold-out is when you split up your dataset into a ‘train’ and ‘test’ set. The training set is what the model is trained on, and the test set is used to see how well that model performs on unseen data. A common split when using the hold-out method is using 80% of data for training and the remaining 20% of the data for testing.
2- k Fold Coss Validation
What is the difference between holdout and cross-validation? The holdout technique is an exhaustive cross-validation method, that randomly splits the dataset into train and test data depending on data analysis. In the case of holdout cross-validation, the dataset is randomly split into training and validation data. Generally, the split of training data is more than test data
3- Bootstrap Validation
Bootstrapping is any test or metric that relies on random sampling with replacement.It is a method that helps in many situations like validation of a predictive model performance, ensemble methods, estimation of bias and variance of the parameter of a model etc.
MODEL EVALUATION METRICS IN MACHINE LEARNING
Model evaluation is important to assess the efficacy of a model during initial research phases
FOR REGRESSION
1-MSE
If the data contains a huge number of outliers, then this metric is known to be a good one.
2- RMSE
This is one of the popular metrics that is mainly used in regression problems. This metric assumes that the error is unbiased and follows a normal distribution.
3- MAE
This metric represents the average of the absolute differences between the actual observation and the prediction.
FOR CLASSIFICATION
1-Confusion Matrix
When performing classification predictions, there’s four types of outcomes that could occur.
- True positives -are when you predict an observation belongs to a class and it actually does belong to that class.
- True negatives -are when you predict an observation does not belong to a class and it actually does not belong to that class.
- False positives-occur when you predict an observation belongs to a class when in reality it does not.
- False negatives-occur when you predict an observation does not belong to a class when in fact it does.
- Accuracy is defined as the percentage of correct predictions for the test data. It can be calculated easily by dividing the number of correct predictions by the number of total predictions.
accuracy=correct predictions/all predictions
- Recall is defined as the fraction of examples which were predicted to belong to a class with respect to all of the examples that truly belong in the class.
recall=truepositivestruepositives+falsenegatives
- Precision is defined as the fraction of relevant examples (true positives) among all of the examples which were predicted to belong in a certain class.
precision=truepositives/truepositives+falsepositives
2-ROC Curve — Receiver Operating Characteristic- AUC
Another common metric is AUC, area under the receiver operating characteristic (ROC) curve. The Reciever operating characteristic curve plots the true positive (TP) rate versus the false positive (FP) rate at different classification thresholds. The thresholds are different probability cutoffs that separate the two classes in binary classification. It uses probability to tell us how well a model separates the classes.
BIAS-VARIANCE TRADEOFF
Bias-variance trade-off is tension between the error introduced by the bias and the error produced by the variance
INCREASING MODEL PERFORMANCE
1- Model Parameters
These are the parameters in the model that must be determined using the training data set. These are the fitted parameters.
2 -Hyperparameters
These are adjustable parameters that must be tuned in order to obtain a model with optimal performance.
Model parameters are internal to the model and estimated from data automatically, whereas Hyperparameters are set manually and are used in the optimization of the model and help in estimating the model parameters.m
3 -Parameters- Hyperparameters Tuning
Let’s start with the difference between parameters and hyperparameters which is extremely important to know. Parameters are the components of the model that are learned during the training process and we can never set them manually. A model starts the training process with random parameter values and adjusts them throughout. Whereas, hyperparameters are the components set by you before the training of the model. The values of hyperparameters might improve or worsen your model’s accuracy.
What is the need for hyperparameter tuning in machine learning? Machine learning models are not intelligent enough to know what hyperparameters would lead to the highest possible accuracy on the given dataset. However, hyperparameter values when set right can build highly accurate models, and thus we allow our models to try different combinations of hyperparameters during the training process and make predictions with the best combination of hyperparameter values. Some of the hyperparameters in Random Forest Classifier are n_estimators (total number of trees in a forest), max_depth (the depth of each tree in the forest), and criterion (the method to make splits in each tree). n_estimators set to 1 or 2 doesn’t make sense as a forest must have a higher number of trees, but how do we know what number of trees will yield the best results? And for this purpose, we try different values like [100, 200, 300]. The model will try all three of the given values and we can easily identify the optimal number of trees in our forest.
Hyperparameter tuning in Python
We have three methods of hyperparameter tuning in python are Grid search, Random search, and Informed search
1- GRID SEARCH
In grid search, each square in a grid has a combination of hyperparameters and the model has to train itself on each combination.
from sklearn.model_selection import GridSearchCV
Example:
grid_df = GridSearchCV(estimator=model, param_grid=grid_vals, scoring=’accuracy’)
2- RANDOM SEARCH
Like grid search, we set the hyperparameter values we want to tune in Random Search. However, the model does not train each combination of hyperparameters, it instead selects them randomly. We have to define the number of samples we want to choose from our grid.
