Bayesian Optimization for Expensive Tuning Problems

See the simulated run below for a full tuning example. It compares Bayesian optimization with random search on the same simulated task.

Worked Example (Simulated): Bayesian Search or Random Search for C?

Question: Can a Bayesian search find a useful value for C with fewer checks than a random search?

Inputs: Simulated data, seed 23. The script makes simulated classification data, then splits the rows for fit, choice, and final check. It tunes C for an RBF support vector classifier. C ranges from 0.01 to 100 on a log scale.

Method: Each search gets the same number of checks, 9. The Bayesian search uses a Gaussian process and expected improvement to pick each next value. The random search checks values drawn from the same range. Both methods use the same fit and choice data. The script checks each best value on the held-out rows.

import warnings
import numpy as np
from scipy.stats import norm
from sklearn.datasets import make_classification
from sklearn.exceptions import ConvergenceWarning
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern, WhiteKernel
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

warnings.filterwarnings("ignore", category=ConvergenceWarning)
SEED = 23
X, y = make_classification(
    n_samples=480, n_features=8, n_informative=5, n_redundant=1,
    n_classes=2, class_sep=0.9, flip_y=0.05, random_state=SEED
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=SEED
)
X_fit, X_valid, y_fit, y_valid = train_test_split(
    X_train, y_train, test_size=0.25, stratify=y_train, random_state=SEED
)

def score(log_c, X_eval, y_eval):
    svc = SVC(C=10.0 ** float(log_c), kernel="rbf", gamma="scale")
    model = make_pipeline(StandardScaler(), svc)
    model.fit(X_fit, y_fit)
    return model.score(X_eval, y_eval)

# Search C in [0.01, 100], represented by log10(C) in [-2, 2].
grid = np.linspace(-2.0, 2.0, 81)
initial_indices = [8, 40, 72]
bo_x = list(grid[initial_indices])
bo_y = [score(x, X_valid, y_valid) for x in bo_x]

for _ in range(6):
    gp = GaussianProcessRegressor(
        kernel=Matern(nu=2.5) + WhiteKernel(noise_level=1e-5),
        normalize_y=True, random_state=SEED, n_restarts_optimizer=0
    )
    gp.fit(np.array(bo_x).reshape(-1, 1), np.array(bo_y))
    remaining = np.array([x for x in grid if not np.any(np.isclose(x, bo_x))])
    mean, std = gp.predict(remaining.reshape(-1, 1), return_std=True)
    improvement = mean - max(bo_y) - 0.01
    z = np.divide(improvement, std, out=np.zeros_like(improvement),
                  where=std > 0)
    ei = improvement * norm.cdf(z) + std * norm.pdf(z)
    next_x = remaining[int(np.argmax(ei))]
    bo_x.append(float(next_x))
    bo_y.append(score(next_x, X_valid, y_valid))

rng = np.random.default_rng(SEED)
random_indices = rng.choice(len(grid), size=9, replace=False)
random_x = grid[random_indices]
random_y = [score(x, X_valid, y_valid) for x in random_x]

bo_best_index = int(np.argmax(bo_y))
random_best_index = int(np.argmax(random_y))
bo_best_x = bo_x[bo_best_index]
random_best_x = random_x[random_best_index]
bo_test = score(bo_best_x, X_test, y_test)
random_test = score(random_best_x, X_test, y_test)

print("Simulated data, seed 23")
print(f"Rows: {len(X)}; features: {X.shape[1]}; "
      f"held-out test rows: {len(y_test)}")
print("Task: maximize validation accuracy for an RBF support vector classifier")
print("Search space: C from 0.01 to 100 on a log scale; "
      "9 evaluations per method")
for name, bx, vy, ty in [
    ("Bayesian optimization", bo_best_x, bo_y[bo_best_index], bo_test),
    ("Random search", random_best_x, random_y[random_best_index], random_test),
]:
    print(f"{name}: best C={10.0 ** bx:.4g}, "
          f"validation accuracy={vy:.3f}, test accuracy={ty:.3f}")

Printed output:

Simulated data, seed 23
Rows: 480; features: 8; held-out test rows: 120
Task: maximize validation accuracy for an RBF support vector classifier
Search space: C from 0.01 to 100 on a log scale; 9 evaluations per method
Bayesian optimization: best C=1, validation accuracy=0.889, test accuracy=0.817
Random search: best C=2.512, validation accuracy=0.878, test accuracy=0.800

Result: Bayesian search chose C=1, with a validation score of 0.889 and a test score of 0.817. Random search chose C=2.512, with a validation score of 0.878 and a test score of 0.800. In this run, the Bayesian search had the higher scores on both sets. Test accuracy 0.817 against 0.800 is a difference of 2 of 120 test rows.

Limit: This is one synthetic task and one split. It cannot show which search will do better on other data.

The Experimental Cost Problem: Why Random Search Can Be Costly for Expensive Tests

Before we discuss solutions, let's examine the problem Bayesian optimization solves. You're tuning a machine learning model with five hyperparameters. A full grid search with just 10 values per parameter requires 100,000 evaluations. At 30 seconds per evaluation, that's 35 days of continuous computation.

Random search improves this by sampling configurations at random rather than exhaustively testing all combinations. It's surprisingly effective: it can often match grid search performance with far fewer evaluations. But it still wastes evaluations on obviously bad configurations and redundantly tests similar parameter settings.

Here's the core inefficiency: neither grid search nor random search learns from experimental results. After running 50 evaluations showing that learning rates above 0.01 consistently fail, these methods happily waste evaluation 51 testing learning rate 0.05.

What Patterns Can a Bayesian Optimization Run Show?

In the simulated run above, Bayesian optimization uses earlier validation scores to guide the next value of C. After each evaluation, it updates a probabilistic model of the objective function: a statistical representation of how parameters affect performance. This model can hint at patterns such as these:

Pattern 1: Regions of Diminishing Returns

Some parameter ranges show flat performance. Increasing training epochs from 50 to 100 might improve accuracy, but 100 to 150 barely changes results. The Gaussian process model detects this diminishing return, redirecting experimental budget toward more sensitive parameters.

Pattern 2: Interaction Effects Between Parameters

In some training setups, learning rate and batch size interact: the best learning rate can shift when the batch size changes, and how much depends on the model, optimizer and data (this is a task-dependent example, not a rule). Grid search covers every combination on its grid, which includes interacting pairs but spends runs on poor ones. Because a Gaussian process models the objective jointly over all parameters, Bayesian optimization can concentrate later evaluations near a good combination if such a ridge exists.

Pattern 3: Multimodal Objective Functions

Sometimes multiple parameter configurations achieve similar performance through different mechanisms. A neural network might work well with many shallow layers OR few deep layers. The acquisition function balances exploiting known good regions with exploring for alternative optima, revealing this hidden multi-peak structure.

Pattern 4: Sensitivity Hierarchies

Not all parameters matter equally. Learning rate might dominate model performance while dropout rate has minimal impact. A Gaussian process with a separate length scale per parameter (automatic relevance determination) can give a rough estimate of this hierarchy: a parameter with a long fitted length scale changes the objective little over its range. The estimate depends on the number of evaluations and the noise, so treat it as a hint to check, for example with partial dependence plots, and not as a measurement.

The Mechanics: How Bayesian Optimization Learns From Experiments

Let's walk through exactly how Bayesian optimization decides which experiment to run next. Understanding this process helps you set up optimization runs correctly and interpret results.

Component 1: The Surrogate Model (Gaussian Process)

After each experiment, Bayesian optimization fits a Gaussian process to all observed results. This probabilistic model provides two critical pieces of information at every point in the parameter space:

• Predicted performance (mean): Based on nearby experiments, what result do we expect?
• Uncertainty (variance): How confident is this prediction? Untested regions have high uncertainty.

The Gaussian process acts as a surrogate for the true (unknown) objective function. It's cheap to evaluate, you can query millions of points instantly, while the real objective requires expensive experiments.

After 10 evaluations tuning learning rate and batch size:

Evaluation results stored:
  lr=0.001, batch=32  → accuracy=0.78
  lr=0.01,  batch=32  → accuracy=0.82
  lr=0.1,   batch=32  → accuracy=0.71
  lr=0.001, batch=128 → accuracy=0.74
  ... (6 more evaluations)

Gaussian process model at untested point (lr=0.005, batch=64):
  Predicted accuracy: 0.80 (± 0.03)

Interpretation:
  - Mean (0.80): Based on nearby evaluations, we expect 80% accuracy
  - Uncertainty (±0.03): We're fairly confident since we've tested similar configs

Compare to unexplored region (lr=0.0001, batch=256):
  Predicted accuracy: 0.76 (± 0.12)

Interpretation:
  - Mean (0.76): Extrapolating from trends, probably lower performance
  - Uncertainty (±0.12): High uncertainty. We haven't tested this region

The high uncertainty makes this region worth exploring. It might contain hidden optima.

Component 2: The Acquisition Function (Experiment Selection Logic)

With predictions and uncertainties in hand, how do we choose the next experiment? This is where acquisition functions enter. They convert the Gaussian process model into a concrete recommendation: "Test this configuration next."

The three most common acquisition functions handle the exploration-exploitation tradeoff differently:

Expected Improvement (EI): The Balanced Explorer

Expected Improvement calculates: "How much better than the current best result could this configuration be, accounting for uncertainty?" It naturally balances:

• Exploitation: Configurations near the current best (high predicted performance)
• Exploration: Uncertain regions that might hide better solutions (high variance)

Expected Improvement (EI) is one of several acquisition functions; its fit depends on the task.

Upper Confidence Bound (UCB): The Optimistic Searcher

UCB evaluates: "predicted performance + (exploration bonus × uncertainty)". The exploration bonus parameter lets you control the exploration-exploitation balance explicitly:

• High bonus (κ=2.5-3.0): Explore aggressively, good for finding global optima in complex spaces
• Low bonus (κ=0.5-1.0): Exploit known good regions, good for fast convergence when you need results quickly

Probability of Improvement (PI): The Conservative Optimizer

PI asks: "What's the probability this configuration beats the current best?" It favors safe bets over risky exploration. Use PI when:

• You have a good baseline and want incremental improvement
• The experimental budget is very limited (10-15 evaluations)
• Local optima are acceptable. You don't need the global optimum

Component 3: The Optimization Loop

Putting it together, Bayesian optimization runs this loop:

Each iteration refines the Gaussian process model, sharpening predictions in explored regions while identifying which unexplored areas warrant investigation. The acquisition function balances exploration and exploitation. With common choices such as expected improvement, the search tends to move from exploring uncertain areas toward refining promising ones as uncertainty shrinks, though this depends on the function and the problem.

Step-by-Step Implementation: From Problem to Solution

Let's walk through a concrete optimization problem to see how you'd set up and run Bayesian optimization in practice.

The Problem: Optimizing Customer Churn Model Performance

You've built a gradient boosting model to predict customer churn. It works, but you suspect better hyperparameter settings could improve accuracy. The model has five key parameters:

• n_estimators: Number of boosting rounds (50-500)
• max_depth: Tree depth (3-12)
• learning_rate: Step size (0.001-0.3)
• min_samples_split: Minimum samples to split a node (2-50)
• subsample: Fraction of samples per tree (0.5-1.0)

Each training run takes 8 minutes on your hardware. Grid search with 10 values per parameter = 100,000 combinations = about 18 months. Even testing 1,000 random combinations takes 5.5 days. You need results this week.

Step 1: Define the Objective Function

The objective function takes hyperparameters as input and returns performance as output. For Bayesian optimization, this function should be:

• Deterministic (or low-noise): Same inputs produce same outputs
• Smooth objective: A smooth objective can suit some surrogate models. For noisy scores or integer and category settings, choose a model and noise setting that fit the task
• Expensive to evaluate: Otherwise just use random search

def objective(n_estimators, max_depth, learning_rate,
               min_samples_split, subsample):
    """
    Train gradient boosting model with given hyperparameters,
    return validation accuracy.
    """
    model = GradientBoostingClassifier(
        n_estimators=int(n_estimators),
        max_depth=int(max_depth),
        learning_rate=learning_rate,
        min_samples_split=int(min_samples_split),
        subsample=subsample,
        random_state=42
    )

    # Train on training set, evaluate on validation set
    model.fit(X_train, y_train)
    accuracy = model.score(X_valid, y_valid)

    # Return negative accuracy (Bayesian opt minimizes by default)
    return -accuracy

Step 2: Specify the Search Space

Define bounds and types for each parameter. This shapes what the optimizer explores:

from skopt.space import Integer, Real

search_space = [
    Integer(50, 500, name='n_estimators'),
    Integer(3, 12, name='max_depth'),
    Real(0.001, 0.3, name='learning_rate', prior='log-uniform'),
    Integer(2, 50, name='min_samples_split'),
    Real(0.5, 1.0, name='subsample')
]

# Note: 'log-uniform' for learning_rate means the optimizer will
# explore 0.001, 0.01, 0.1 roughly equally rather than focusing
# on larger values. Use this for parameters that span orders of magnitude.

Step 3: Configure Bayesian Optimization

Set up the optimization run with appropriate initial samples and acquisition function:

Configuration decisions:

1. Number of random initializations: 10
   (5 parameters × 2 = 10 samples to seed the Gaussian process)

2. Total budget: 50 evaluations
   (10 random + 40 Bayesian-guided evaluations)
   Expected time: 50 × 8 min = 6.7 hours

3. Acquisition function: Expected Improvement (EI)
   (Balances exploration and exploitation)

4. Gaussian process kernel: Matern 5/2
   (Assumes a smooth objective)

5. Evaluation parallelization (optional): 4 parallel evaluations
   (If you have 4 GPUs/cores available)
   gp_minimize below evaluates one configuration at a time. The ask/tell interface can request multiple points at once.
   Check the framework docs for its current use.
   Reduces wall-clock time to roughly: 50/4 × 8 min = 1.7 hours

Step 4: Run Optimization and Monitor Progress

Execute the optimization loop. Modern frameworks handle the Gaussian process fitting and acquisition function optimization internally. You just feed in results:

from skopt import gp_minimize
from skopt.plots import plot_convergence, plot_objective

# Run Bayesian optimization
result = gp_minimize(
    func=objective,
    dimensions=search_space,
    n_calls=50,              # Total evaluations
    n_initial_points=10,     # Random initializations
    acq_func='EI',           # Expected Improvement
    n_jobs=4,                # Cores for optimizing the acquisition function;
                             # objective evaluations still run one at a time
    random_state=42,
    verbose=True
)

# Best configuration found
best_params = result.x
best_accuracy = -result.fun  # Negate back to accuracy

print(f"Best accuracy: {best_accuracy:.4f}")
print(f"Best hyperparameters:")
print(f"  n_estimators: {best_params[0]}")
print(f"  max_depth: {best_params[1]}")
print(f"  learning_rate: {best_params[2]:.4f}")
print(f"  min_samples_split: {best_params[3]}")
print(f"  subsample: {best_params[4]:.3f}")

# Visualize convergence
plot_convergence(result)

Step 5: Validate and Analyze Results

Before declaring victory, validate the found optimum and understand what the search revealed:

Interpreting Results: What the Model Reveals About Your Search Space

The Gaussian process model fitted during optimization contains valuable insights beyond just "use these parameter values." Here's how to extract that knowledge.

Convergence Plots: Did We Find the Optimum?

Plot the best-observed value versus evaluation number. You should see rapid improvement in the first 10-20 evaluations as the optimizer locates promising regions, then gradual refinement as it exploits the best area.

Pattern 1: Plateau after 30 evaluations
  → Optimization converged, found the optimum (or local optimum)
  → Additional evaluations unlikely to help
  → Interpretation: The search space is well-understood

Pattern 2: Still improving at evaluation 50
  → Haven't found the optimum yet
  → Run more evaluations (extend to 75-100)
  → Interpretation: Complex search space or multiple optima

Pattern 3: Noisy, no clear trend
  → Objective function has high variance (randomness in evaluation)
  → Consider averaging multiple evaluations per configuration
  → Or use noise-robust acquisition functions (Expected Improvement with noise)

Pattern 4: Step changes at specific evaluations
  → Optimizer discovered a new promising region
  → Check which parameter values changed at those evaluations
  → Interpretation: Non-smooth objective or interaction effects

Partial Dependence: Which Parameters Actually Matter?

Partial dependence plots show how performance changes with each parameter while marginalizing over others. This shows parameter sensitivity and promising ranges:

• Flat curve: Parameter doesn't matter much, any value in the range works
• Sharp peak: Parameter is sensitive, optimal value is narrow (must be tuned precisely)
• Monotonic trend: Performance consistently improves/degrades with the parameter (bound might be too restrictive)
• U-shaped curve: Extreme values work, middle values don't, or vice versa

Interaction Effects: Parameter Dependencies

Two-dimensional partial dependence plots show how pairs of parameters interact. Look for:

• Diagonal ridge patterns: Parameters should be tuned together (e.g., learning_rate and n_estimators often compensate for each other)
• Independent hotspots: Parameters don't interact, can be tuned separately
• Multiple optimal regions: Different parameter combinations achieve similar performance through different mechanisms

Acquisition Function Surface: Where Did We Explore?

Plotting the acquisition function value across the search space shows where the optimizer decided to explore and why:

• High acquisition value: Regions worth exploring (uncertain or potentially better than current best)
• Low acquisition value: Regions already well-understood or clearly suboptimal
• Evaluated points: Show which configurations were actually tested

This reveals whether the optimizer thoroughly explored the space or got stuck exploiting a local optimum.

Best Practices: Running Effective Bayesian Optimization

Decades of research and practical application have identified patterns that separate successful optimizations from failed attempts. Follow these guidelines.

Match the Algorithm to the Experiment Cost

Bayesian optimization's value proposition scales with evaluation expense:

Choose Initial Samples Wisely

The random initialization phase seeds the Gaussian process. Too few samples leave the model uninformed; too many waste budget on random exploration.

Use these heuristics:

• Low-dimensional (1-3 parameters): 3-5 random samples
• Medium-dimensional (4-7 parameters): 8-15 random samples (roughly 2× dimension)
• High-dimensional (8+ parameters): 15-25 random samples, or consider dimensionality reduction first

If you have prior knowledge (e.g., reasonable defaults), include those as initial samples rather than purely random.

Set Realistic Parameter Bounds

The search space boundaries strongly influence optimization effectiveness. Bounds that are too wide waste evaluations exploring clearly bad regions; bounds too narrow might exclude the optimum.

Bad bounds (too wide):
  learning_rate: [1e-10, 1.0]
  → Wastes evaluations on 1e-10, 1e-9, 1e-8 which are all equivalently bad

Good bounds (informed by domain knowledge):
  learning_rate: [1e-4, 0.5]
  → Focuses on plausible range where optimum likely exists

Bad bounds (too narrow):
  n_estimators: [90, 110]
  → If you already know optimal is ~100, just use 100 directly

Good bounds (captures uncertainty):
  n_estimators: [50, 300]
  → Allows discovering whether 100 is actually optimal or just conventional

For parameters spanning orders of magnitude:
  Use log-uniform sampling: learning_rate ~ log-uniform(1e-4, 1e-1)
  This explores 1e-4, 1e-3, 1e-2, 1e-1 roughly equally

Handle Noisy Objectives Appropriately

If your objective function has intrinsic randomness (stochastic training, A/B test variance, manufacturing variability), the Gaussian process needs to account for this noise. Otherwise, it overfits to random fluctuations.

Strategies for noisy objectives:

• Model observation noise: Use Gaussian process with noise parameter (automatically learns noise level from data)
• Multiple evaluations per configuration: Average 3-5 runs of stochastic objectives to reduce variance
• Noise-robust acquisition functions: Expected Improvement naturally handles some noise; Knowledge Gradient explicitly optimizes for noisy settings
• Increase exploration: Higher UCB κ values (2.5-3.0) prevent premature convergence due to noise

Monitor Convergence, Don't Just Hit Budget

The "optimal" number of evaluations isn't predetermined. Monitor these signals:

• No improvement in 10 consecutive evaluations: Likely converged, additional evaluations have low marginal value
• Acquisition function values all low: Every candidate point looks unpromising, search space is well-understood
• Gaussian process uncertainty collapsed everywhere: Model is confident about predictions across the space

Conversely, if you hit your evaluation budget while still seeing steady improvement, extend the budget. You haven't found the optimum yet.

Validate on Independent Data

The configuration that performed best during optimization might be overfit to your validation set, especially if you ran many evaluations. Always:

1. Retrain from scratch with the "optimal" hyperparameters
2. Evaluate on test data not used during optimization
3. Run multiple replicates (3-5) if there's training randomness
4. Compare to baseline with statistical significance test

If test performance significantly underperforms optimization performance, you overfit. Consider using a smaller fraction of data for validation during optimization to reduce overfitting risk.

Related Optimization Techniques

Bayesian optimization is powerful, but it's one tool in a broader optimization toolkit. Understanding alternatives helps you choose the right method for each problem.

Hyperparameter Tuning: Systematic Search Methods

Hyperparameter tuning encompasses multiple approaches beyond Bayesian optimization. Grid search exhaustively tests all combinations, random search samples randomly, and Bayesian optimization learns from results. The linked guide compares all three approaches with decision criteria for when to use each.

Multi-Fidelity Optimization: Faster Feedback Loops

Multi-fidelity methods (like Hyperband, BOHB) recognize that early stopping provides cheap, informative signals. Instead of fully training every configuration, they:

• Start many configurations with minimal training
• Eliminate clearly bad ones after a few epochs
• Allocate most budget to promising configurations

Use multi-fidelity when evaluations are progressive (can be stopped early) and partial evaluation correlates with full evaluation. It reduces total compute 3-10× compared to standard Bayesian optimization.

Gradient-Based Optimization: When Derivatives Are Available

If your objective function is differentiable with respect to hyperparameters (rare but possible in some neural network settings), gradient-based methods converge much faster than Bayesian optimization. Techniques like implicit differentiation or backpropagation through the optimization process enable this.

Trade-off: Requires differentiable objective and careful implementation. Most hyperparameter tuning problems don't have accessible gradients.

Evolutionary Algorithms: Population-Based Search

Genetic algorithms and evolution strategies maintain a population of configurations, mutating and recombining promising ones. They handle discrete, categorical, and highly multimodal objectives well.

Use evolutionary approaches when:

• Objective is discontinuous or has many local optima
• You can evaluate many configurations in parallel (100+)
• Parameter space includes complex structures (e.g., neural architecture search)

Trade-off: Requires larger total budget than Bayesian optimization but parallelizes better.

Conclusion: Intelligent Experimentation Over Brute Force

Bayesian optimization transforms how we approach expensive search problems. By building a probabilistic model of the objective function and intelligently selecting which experiments reveal the most information, it can sometimes reach a good configuration in fewer evaluations than random or grid search.

When evaluations are expensive, that efficiency can translate into real value:

• Less optimization time for hyperparameter tuning
• Lower cloud compute costs for large-scale model training
• Faster time-to-production for optimized models
• Often better final performance for a fixed budget, through targeted exploration of the search space

The method's success depends on proper experimental design: defining realistic parameter bounds, choosing appropriate acquisition functions, handling noise correctly, and validating on independent data. When applied rigorously to expensive optimization problems, Bayesian optimization often delivers strong results at a fraction of the cost of uninformed search.

The patterns an optimization run can hint at (parameter sensitivity hierarchies, interaction effects, multimodal optima, diminishing return regions) can offer clues beyond finding good settings. Check them before you rely on them; they can inform future modeling choices and deepen understanding of how your system behaves.

Frequently Asked Questions

What makes Bayesian optimization different from grid search?

Grid search evaluates every possible combination blindly, testing hundreds or thousands of configurations. Bayesian optimization learns from each experiment: it builds a probabilistic model of which configurations work well, then intelligently selects the next most informative experiment. On expensive problems this can cut the number of evaluations from hundreds to tens, with compute cost falling in proportion, and it often finds solutions as good or better.

When should I use Bayesian optimization instead of random search?

Use Bayesian optimization when each experiment is expensive in time, money, or resources. If a single evaluation takes more than 30 seconds, costs significant compute resources, or requires running a full production test, Bayesian optimization pays off. Random search works fine for cheap evaluations where you can afford thousands of evaluations. Random search is simpler to set up, so compare the two on your own problem.

What is an acquisition function and why does it matter?

An acquisition function decides which experiment to run next. Expected Improvement (EI) balances exploring uncertain regions with exploiting known good areas. Probability of Improvement (PI) focuses on beating the current best. Upper Confidence Bound (UCB) explicitly trades off exploration and exploitation with a tunable parameter. The choice affects whether you find the global optimum (EI, UCB) or converge quickly to a good-enough solution (PI).

How many initial random samples do I need before optimization starts?

Start with 3-5 random samples for 1-2 parameters, 5-10 samples for 3-5 parameters, and 10-15 samples for 6+ parameters. These random samples establish a baseline understanding of the search space before the Gaussian process model takes over. Too few samples give the model insufficient information; too many waste budget on random exploration. The rule of thumb is 2-3x the number of parameters being optimized.

Can Bayesian optimization handle categorical parameters like algorithm choice?

Yes, modern Bayesian optimization frameworks handle mixed parameter spaces: continuous (learning rate 0.001-0.1), integer (number of layers 1-10), and categorical (optimizer: Adam, SGD, RMSprop). The Gaussian process uses specialized kernels for categorical variables. This lets you optimize not just numeric parameters but also discrete choices like which algorithm, activation function, or feature engineering method to use.

About Bayesian Optimization

Bayesian optimization is used to tune settings when each evaluation is expensive, such as model training or lab runs.