GaussianHyperparameterSearch

Dictionary containing numerical arrays of length two holding the lower and upper bounds of all parameters which will be altered in pipeline during the hyperparameter optimization.

If we have two feature learners and one predictor, the hyperparameter space might look like this:

param_space = {
    "feature_learners": [
        {
            "num_features": [10, 50],
        },
        {
            "max_depth": [1, 10],
            "min_num_samples": [100, 500],
            "num_features": [10, 50],
            "reg_lambda": [0.0, 0.1],
            "shrinkage": [0.01, 0.4]
        }],
    "predictors": [
        {
            "reg_lambda": [0.0, 10.0]
        }
    ]
}

If we only want to optimize the predictor, then we can leave out the feature learners.

Base pipeline used to derive all models fitted and scored during the hyperparameter optimization. Be careful when constructing it since only the parameters present in param_space will be overwritten. It defines the data schema and any hyperparameters that are not optimized.

The score to optimize. Must be from metrics.

Number of iterations in the hyperparameter optimization and thus the number of parameter combinations to draw and evaluate. Range: [1, ∞]

Seed used for the random number generator that underlies the sampling procedure to make the calculation reproducible. Due to nature of the underlying algorithm, this is only the case if the fit is done without multithreading. To reflect this, a seed of None is only allowed to be set to an actual integer if both num_threads and n_jobs instance variables of the predictor and feature_selector in model - if they are instances of either XGBoostRegressor or XGBoostClassifier - are set to 1. Internally, a seed of None will be mapped to 5543. Range: [0, ∞]

Ratio of the iterations used for the burn-in. For a ratio_iter of 1.0, all iterations will be spent in the burn-in period resulting in an equivalence of this class to LatinHypercubeSearch or RandomSearch - depending on surrogate_burn_in_algorithm. Range: [0, 1]

As a rule of thumb at least 70 percent of the evaluations should be spent in the burn-in phase. The more comprehensive the exploration of the param_space during the burn-in, the less likely it is that the Gaussian process gets stuck in local minima.

Determines the optimization algorithm used for the local search in the optimization of the expected information (EI). Must be from optimization.

Specifies the algorithm used to draw initial points in the burn-in period of the optimization of the expected information (EI). Must be from burn_in.

Number of random evaluation points used during the burn-in of the minimization of the expected information (EI). After the surrogate model - the Gaussian process - was successfully fitted to the previous parameter combination, the algorithm is able to calculate the EI for a given point. In order to get to the next combination, the EI has to be maximized over the whole parameter space. Much like the GaussianProcess itself, this requires a burn-in phase. Range: [3, ∞]

Specifies the algorithm used to draw new parameter combinations during the burn-in period. Must be from burn_in.

Specifies the 1-dimensional kernel of the Gaussian process which will be used along each dimension of the parameter space. All the choices below will result in continuous sample paths and their main difference is the degree of smoothness of the results with 'exp' yielding the least and 'gauss' yielding the most smooth paths. Must be from kernels.

Specifies the algorithm used to draw new parameter combinations during the burn-in period of the optimization of the Gaussian process. Must be from burn_in.

Determines the optimization algorithm used for the local search in the fitting of the Gaussian process to the previous parameter combinations. Must be from optimization.

Number of random evaluation points used during the burn-in of the fitting of the Gaussian process. Range: [3, ∞]

Whether you want to apply early stopping to the predictors.