Hyperparameters in Machine Learning

Making your Models Perform

Machine learning algorithms are powerful tools that uncover hidden patterns in data, but their true potential is unlocked through careful configuration.

These algorithms aren’t static entities; they come with adjustable settings that significantly influence their learning process and ultimately, their performance. These settings are known as hyperparameters.

Think of a machine learning algorithm as a sophisticated recipe. The data are your ingredients, and the algorithm is the cooking method. Hyperparameters are like the adjustable knobs on your oven (temperature, cooking time) or the specific measurements you choose to add ingredients. Setting them correctly is crucial for achieving the desired dish — a well-performing model.

Unlike the model’s internal parameters (the weights and biases learned during training), hyperparameters are set before the training process begins. They govern the structural aspects of the model and the optimization strategy. Choosing the right hyperparameters can drastically impact a model’s accuracy, training speed, and ability to generalize. This often requires experimentation and a solid understanding of the algorithm.

In this post, we will explore key hyperparameters in popular machine learning algorithms and discuss best practices for tuning them effectively.

Why Hyperparameters Matter

Hyperparameters influence:

• Model Complexity (e.g., tree depth in Decision Trees)
• Regularization (e.g., preventing overfitting in Logistic Regression)
• Distance Metrics (e.g., in K-Nearest Neighbors)
• Convergence Speed (e.g., learning rate in Neural Networks)

Poor hyperparameter choices can lead to underfitting, overfitting, or inefficient training. Let’s examine key examples across different algorithms.

Key Hyperparameters in Popular Algorithms

1. Linear Regression

While often considered a simpler algorithm, Linear Regression benefits from hyperparameters when dealing with multicollinearity or the risk of overfitting.

a. Regularization Parameter (alpha for Ridge/Lasso Regression):

• Concept: Regularization techniques like Ridge (L2) and Lasso (L1) add a penalty term to the cost function to shrink the model’s coefficients. This helps prevent the model from becoming too complex and fitting the noise in the training data.
• alpha (in scikit-learn): This hyperparameter controls the strength of the regularization.

i. A higher alpha increases the penalty, leading to smaller coefficients and a simpler model, which can help with overfitting but might underfit if set too high.

ii. A lower alpha reduces the penalty, making the model more flexible and potentially leading to overfitting if not carefully managed.

from sklearn.linear_model import Ridge

ridge_model = Ridge(alpha=1.0)  # Regularization strength        

2. Logistic Regression

Used for binary and multi-class classification, Logistic Regression also employs regularization to improve its generalization ability.

a. C (Inverse of Regularization Strength):

• Concept: Similar to alpha in linear regression, C controls the regularization strength. However, C is the inverse of the regularization parameter.
• A higher C means weaker regularization, allowing the model to fit the training data more closely, potentially leading to overfitting.
• A lower C means stronger regularization, forcing the model to have smaller coefficients and potentially underfitting.

b. penalty (L1, L2):

• Concept: Specifies the type of regularization to be applied.
• L1 (Lasso): Can drive some feature coefficients to exactly zero, effectively performing feature selection.
• L2 (Ridge): Shrinks coefficients towards zero but rarely makes them exactly zero.

from sklearn.linear_model import LogisticRegression

log_reg = LogisticRegression(C=0.5, penalty='l2')  # L2 regularization        

3. Decision Tree

Decision Trees learn by recursively splitting the data based on feature values. Hyperparameters control the structure and complexity of these trees.

a. max_depth: The maximum depth of the tree. A deeper tree can capture more complex relationships but is more prone to overfitting.

b. min_samples_split: The minimum number of samples required to split an internal node. Higher values prevent the creation of very specific splits based on small subsets of data.

c.min_samples_leaf: The minimum number of samples required to be at a leaf node. Similar to min_samples_split, this helps prevent the tree from becoming too sensitive to individual data points.

d.criterion: The function used to measure the quality of a split (e.g., 'gini' for Gini impurity or 'entropy' for information gain in classification).

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(max_depth=5, min_samples_split=10, criterion='entropy')        

4. K-Nearest Neighbors (KNN)

KNN is a non-parametric algorithm that classifies or regresses data points based on the majority class or average value of their nearest neighbors.

a. n_neighbors: The number of neighboring data points to consider when making a prediction.

• A small n_neighbors can make the model sensitive to noise in the data.
• A large n_neighbors can smooth the decision boundaries but might miss local patterns.

b. weights: The weight assigned to each neighbor.

• ‘uniform’: All neighbors are weighted equally.
• ‘distance’: Neighbors closer to the query point have a greater influence.

c. metric: The distance metric to use (e.g., 'euclidean', 'manhattan', 'minkowski'). The choice of metric can significantly impact the results depending on the data distribution.

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier(n_neighbors=5, weights='distance', metric='euclidean')        

5. Support Vector Machines (SVM)

SVMs aim to find the optimal hyperplane that separates different classes or predicts a continuous value.

a. C (Regularization Parameter): Similar to Logistic Regression, C controls the trade-off between achieving a low training error and a low testing error (generalization).

• A high C tries to classify all training examples correctly, potentially leading to a complex model and overfitting.
• A low C allows some misclassifications to achieve a simpler, more generalizable model.

b. kernel: Specifies the kernel function to use. Different kernels allow SVMs to model non-linear relationships (e.g., ‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’).

c. gamma: Kernel coefficient for 'rbf', 'poly', and 'sigmoid'. It influences the reach of a single training example.

• A high gamma means each training example has a local influence, potentially leading to overfitting.
• A low gamma means each training example has a wider influence, potentially leading to underfitting.

d. degree (for polynomial kernel): The degree of the polynomial kernel function.

from sklearn.svm import SVC

svm_model = SVC(C=1.0, kernel='rbf', gamma='scale')        

How to Tune Hyperparameters

1. Grid Search

• Tests all combinations (e.g., C=[0.1, 1, 10] and penalty=['l1','l2']).
• Best for small hyperparameter spaces.

2. Random Search

• Randomly samples combinations (more efficient than Grid Search).

3. Bayesian Optimization

• Uses past evaluations to predict optimal settings (great for expensive models).

4. Automated Tools

• Libraries like Optuna, HyperOpt, and Scikit-learn’s HalvingGridSearchCV optimize tuning.

Best Practices

✅ Start with Defaults (Scikit-learn’s defaults are often reasonable).

✅ Use Cross-Validation (Avoid overfitting with KFold or StratifiedKFold).

✅ Prioritize Impactful Hyperparameters (e.g., n_neighbors in KNN matters more than weights).

✅ Log Experiments (Track performance with tools like MLflow or Weights & Biases).

Conclusion

Hyperparameter tuning is a critical step in building effective machine learning models. Understanding how key hyperparameters like C in SVM, max_depth in Decision Trees, or alpha in Ridge Regression affect performance will help you make informed choices.