Benchmark

Warning

This interface documentation describes a deprecated version of the Analysis and Optimization Toolkit in JCMsuite. It has been superseded by JCMoptimizer and is retained solely for reference in legacy projects.

For all new simulation and optimization projects, use JCMOptimizer instead. The corresponding documentation is available here.

Note

A benchmark object can be created by calling jcmwave.optimizer.create_benchmark().

class jcmwave.client.Benchmark(host, benchmark_id, session, num_average)

This class provides methods for benchmarking different optimization studies against each other. Example:

benchmark = Benchmark(num_average=6)
benchmark.add_study(study1)
benchmark.add_study(study2)
benchmark.set_objective(objective)
benchmark.run()
data = benchmark.get_data(x_type='num_evaluations',y_type='objective',
           average_type='mean')
fig = plt.figure(figsize=(8,4))
for idx,name in enumerate(data['names']):
    X = data['X'][idx]
    Y = np.array(data['Y'][idx])
    std_error = np.array(data['sdev'][idx])/np.sqrt(6)
    p = plt.plot(X,Y,linewidth=2.0, label=name)
    plt.fill_between(X, Y-std_error, Y+std_error, alpha=0.2, color = p[0].get_color())
plt.legend(loc='upper right',ncol=1)
plt.grid()
plt.ylim([0.1,10])
plt.rc('font',family='serif')
plt.xlabel('number of iterations',fontsize=12)
plt.ylabel('average objective',fontsize=12)
plt.show()

add_study(study)

Adds a study to the benchmark. Example:

benchmark.add_study(study1)

Parameters:study – A Study() object.

add_study_results(study)

Adds the results of a benchmark study at the end of an optimization run. Example:

benchmark.add_study_results(study1)

Parameters:study – A Study() object after the study was run.

get_data(**kwargs)

Get benchmark data. Example:

data = benchmark.get_data( x_type='num_evaluations', y_type='objective',
     average_type='mean')
plt.plot(data['X'][0],data['Y'][0])

Parameters:

• x_type (str) – Data on x-axis. Can be either ‘num_evaluations’ or ‘time’
• y_type (str) – Data type on y-axis. Can be either ‘objective’, ‘distance’, (i.e. accumulated minimum distance off all samples to overall minimum), or ‘min_distance’ (i.e. distance of current minimum to overall minimum).
• average_type (str) – Type of averaging over study runs. Can be either ‘mean’ w.r.t. x-axis data or ‘median’ w.r.t. y-axis data
• invert (bool) – If True, the objective is multiplied by -1. (Parameter not available for distance average types)
• log_scale (bool) – If True, the output of Y and sdev are determined as mean and standard deviations of the natural logarithm of the considered y_type.
• minimum (list) – Vector with minimum position. (Only available for distance average types)
• scales (list) – Vector with positive weights for scaling distance in different directions. (Only available for distance average types)
• norm (str/int) – Order of distance norm as defined in numpy.linalg.norm. (Only available for distance average types)
• num_samples (int) – Number of samples on y-axis. (Only available for median average type or time on x-axis)

run()

Run the benchmark after the objective has been set (see set_objective()). Example:

benchmark.run()

set_objective(objective)

Set the objective function to be minimized. Example:

def objective(x1,x2): 
    observation = study.new_observation()
    observation.add(x1**2+x2**2)
    return observation
benchmark.set_objective(objective)

Note

Call this function only after all studies have been added to the benchmark.

Parameters:objective (func) – Function handle for a function of the variable parameters that returns a corresponding Observation() object.

property studies

A list of studies to be run for the benchmark.