Docstrings

AbstractAcquisition

Supertype of the acquisition functions used to form the surrogate model that the sub-optimizer solves. The surrogate objective is minimized, so acquisition values are negated where necessary (EI/PI).

AbstractGPSurrogate

Supertype of the zero-order (ZeroOrderGPSurrogate) and first-order (FirstOrderGPSurrogate) Gaussian-process surrogates.

BayesOptAlg(sub_alg)

Bayesian-optimization meta-algorithm. Expensive functions (flagged with :expensive) are replaced by GP surrogates, whose acquisition model is solved by sub_alg to propose the next query point. The true functions are evaluated at the candidate to update the surrogates.

BayesOptOptions(sub_options; kwargs...)

Options of the BayesOptAlg Bayesian-optimization loop. sub_options (the options of the sub-optimizer) is required. The keyword arguments are:

• maxiter = 100: the maximum number of BO iterations (each iteration evaluates the expensive functions once).
• initialize = true, ninit = 10, init_strategy = :sobol_direct: initialization control; :sobol_direct evaluates the true functions at the Sobol points, :acquisition solves the surrogate from each point.
• ctol = 1e-4: feasibility tolerance; ftol = 1e-4: relative objective tolerance for convergence.
• postoptimize = true: polish the incumbent on the exact model after the loop.
• verbose = false, keep_all = false: per-iteration logging and query trajectory recording.
• acq_multistart = 1, deduplicate = true, xtol_dup = 1e-8: acquisition multistart and duplicate-point filtering.
• trust_region = false, tr_length_init = 0.8, tr_min_factor = 1/32, tr_success_max = 3, tr_fail_max = nothing: TuRBO-style local trust-region restarts (requires finite variable bounds).
• acquisition = nothing: the objective acquisition (default LCB(std_multiple)); constraint surrogates always use LCB.
• std_multiple = 3.0: the η of the default LCB acquisition.
• first_order = false, grad_source = :forwarddiff, fit_grads = true: first-order (gradient-enhanced) surrogates and gradient-observation collection.
• kernel = SqExponentialKernel(), noise = 1e-8: the GP kernel and the observation noise.
• fit_prior = true, fit_noise = false, every = 2, skip = 2, last = :all: hyperparameter refit control.
• fit_max_iter = 20, fit_bounds = nothing, fit_optimizer = nothing: the hyperparameter optimizer (defaults to Ipopt).

BayesOptResult

The result returned by BayesOptAlg. In addition to the GenericResult-style fields (minimizer, minimum, iter, maxiter_reached, tol, convstate, fcalls), it carries the sub_result of the last surrogate solve, the trained surrogates, and the query trajectory when keep_all = true.

• fcalls counts the expensive-function evaluations performed during this optimize call (the initial point is not counted).
• n_restarts counts the TuRBO-style trust-region restarts (0 when trust_region = false).

BayesOptWorkspace

The workspace of the BayesOptAlg meta-algorithm: the original model, the surrogate sub-workspace, the surrogates, the callback and the query history. NonconvexCore.reset! rebuilds the surrogates from scratch so the workspace can be re-run or nested in other meta-algorithms.

CoKrigingSurrogate(f, x; kwargs...)

A co-kriging (gradient-enhanced) Gaussian process surrogate for the expensive function f, initialized at x. Values and gradient observations are conditioned jointly through the kernel-derivative cross-covariances (see CoKrigingPosterior), so the gradient predictions are the exact derivatives of the value predictions: the surrogate is self-consistent, and gradient data sharpens the value model itself.

Gradient observations are collected at every exact-mode evaluation using grad_source (:forwarddiff by default, :finite, :none or a user gradient oracle), as for FirstOrderGPSurrogate. All keyword arguments of ZeroOrderGPSurrogate are supported, plus:

• grad_source = :forwarddiff,
• grad_noise = 1e-8: the jitter on the gradient observations (gradients from AD are treated as exact up to this jitter).

The hyperparameters (kernel, constant mean, optionally the observation noise) are fitted by maximum likelihood on the value observations; the gradient observations then condition the joint posterior under those hyperparameters.

EI(ξ = 0.01)

Negative expected improvement. ξ biases the acquisition towards more exploration.

FirstOrderGPSurrogate(f, x; kwargs...)

A gradient-enhanced (first-order) Gaussian process surrogate for the expensive function f. In addition to the zero-order value GPs it fits one GP per input dimension (per output) on gradient observations, so that first-order sub-optimizers receive the predicted gradient via the ChainRules rule instead of differentiating the value GP — the standard gradient-enhanced (co-kriging style) approach when gradient observations are available. Note that this is an inexact-gradient scheme: the supplied gradient is the gradient-GP prediction and need not equal the AD gradient of the surrogate value.

Gradient observations are collected at every exact-mode evaluation using grad_source:

• :forwarddiff (default): ForwardDiff.jacobian of f,
• :finite: central finite differences (costs 2n extra evaluations per observation),
• :none: no gradient observations; the predicted gradient falls back to the AD gradient of the value GP mean,
• a function ∇f(x): a user-supplied gradient oracle returning a vector (or a matrix with one row per output).

All keyword arguments of ZeroOrderGPSurrogate are supported, plus:

• grad_source = :forwarddiff,
• fit_grads = true: whether to refit the gradient GPs' hyperparameters.

GPUCB(β = 3.0; schedule = nothing)

GP-UCB acquisition μ(x) - β(t) σ(x) where t is the number of observations collected so far and schedule(t) optionally grows β with t (e.g. t -> sqrt(2log(t))) to guarantee no-regret behaviour. Without a schedule, β is constant.

LCB(η = 3.0)

Lower confidence bound acquisition μ(x) - η σ(x). Larger η means more exploration, smaller η more exploitation.

PI(ξ = 0.01)

Negative probability of improvement.

ZeroOrderGPSurrogate(f, x; kwargs...)

A Gaussian process surrogate for the (expensive) function f, initialized at x. In :exact mode it evaluates f and records the observation; in :interval mode it returns an interval built from the acquisition function (by default LCB: μ(x) ± η σ(x)), whose gradient is supplied through a ChainRules rule.

Keyword arguments:

• kernel = SqExponentialKernel(): the GP kernel,
• X = [x], y = [f(x)]: initial observations,
• noise = 1e-8: observation noise,
• std_multiple = 3.0: η of the default LCB acquisition,
• acquisition = nothing: acquisition function; defaults to LCB(std_multiple),
• mode = :interval: :interval or :exact,
• every = 5, skip = 10: refit the GP hyperparameters every every observations after the first skip * every observations,
• last = :all: only use the last last observations when refitting,
• fit_prior = true: whether to refit the hyperparameters by maximum likelihood,
• fit_noise = false: whether to include the noise in the MLE,
• fit_max_iter = 20: maximum iterations of the hyperparameter optimizer,
• fit_bounds = nothing: optional (lb = ..., ub = ...) bounds (scalars or vectors) on the hyperparameters,
• fit_optimizer = nothing: optional (alg, options) tuple used for the hyperparameter optimization; defaults to Ipopt.

co_kriging_surrogate(f, x; kwargs...)

Constructs a CoKrigingSurrogate for f initialized at x (evaluating f(x) and its gradient once) in interval mode.

first_order_surrogate(f, x; kwargs...)

Constructs a FirstOrderGPSurrogate for f initialized at x (evaluating f(x) and its gradient once) in interval mode.

get_lo(x)

The lower bound of an interval, mapped elementwise over arrays and vectors; the identity for reals. Used to extract the acquisition lower bound from a surrogate's interval output.

surrogate(f, x; kwargs...)

Constructs a ZeroOrderGPSurrogate for f initialized at x (evaluating f(x) once) in interval mode.

surrogate_model(vecmodel::VecModel; first_order = false, acquisition = nothing,
                std_multiple = 3.0, kwargs...)

Replaces every expensive objective/constraint function in vecmodel by a GP surrogate, returning (surrogate_model, surrogates). The objective surrogate uses acquisition (default LCB(std_multiple)); constraint surrogates always use LCB(std_multiple) since feasibility calls for a conservative lower bound. When first_order = true, FirstOrderGPSurrogates are used so first-order sub-optimizers receive predicted gradients.

update_surrogates!(model, surrogates, x)

Evaluates model at x with the surrogates in exact mode (recording the true observations, including gradient observations for first-order surrogates), then switches them back to interval mode. Returns the objective value, the inequality and the equality constraint values at x.