Benchmark

Warning

This interface documentation describes a deprecated version of the Analysis and Optimization Toolkit in JCMsuite. It has been superseded by JCMoptimizer and is retained solely for reference in legacy projects.

For all new simulation and optimization projects, use JCMOptimizer instead. The corresponding documentation is available here.

This class provides methods for benchmarking different
optimization studies against each other
Example:

   benchmark = Benchmark('num_average',6);
   benchmark.add_study(study1);
   benchmark.add_study(study2);
   studies = benchmark.studies();
   for i_study = 1:length(studies)
       study = studies{i_study};
       while(not(study.is_done))
           suggestion = study.get_suggestion();
           observation = evaluate(suggestion.sample);
           study.add_observation(observation,suggestion.id);
        end
        benchmark.add_study_results(study);
   end
   data = benchmark.get_data('x_type','num_evaluations','y_type','objective',...
                 'average_type','mean');
   plots = [];
   for ii=1:length(data.names)
        X = data.X(ii,:);
        Y = data.Y(ii,:);
        plots(end+1) = plot(X,Y,'LineWidth',1.5);
   end
   legend(plots, data.names{:});


Methods:

    add_study

        Purpose: Adds a study to the benchmark.
        Usage: benchmark.add_study(study1);
        Input:
           study: A study object.

    add_study_results

        Purpose: Adds the results of a benchmark study at the end
            of an optimization run
        Usage: benchmark.add_study_results(study1);
        Input: A study object after the study was run

    studies

        Purpose: Returns a list of studies to be run for the benchmark
        Usage: studies = benchmark.studies()

    get_data

        Purpose: Get benchmark data
        Usage: data = benchmark.get_data('x_type','num_evaluations',...
                      'y_type','objective','average_type','mean');
        Input:
           x_type: Data on x-axis. Can be either 'num_evaluations' or 'time'
           y_type: Data type on y-axis. Can be either 'objective', 'distance',
               (i.e. accumulated minimum distance off all samples to overall
               minimum), or 'min_distance' (i.e. distance of current minimum
               to overall minimum).
           y_type: Data type on y-axis. Can be either 'objective' or
               'distance', i.e. distance to minimum.
           average_type: Type of averaging over study runs. Can be either
               'mean' w.r.t. x-axis data or 'median' w.r.t. y-axis data
           invert: If True, the objective is multiplied by -1.
               (Parameter not available for distance average types)
           log_scale: If True, the output of Y and sdev are determined as
               mean and standard deviations of the natural logarithm of the
               considered y_type.
           minimum: Vector with minimum position. (Only available for
                distance average types)
           scales: Vector with positive weights for scaling distance in
                different directions. (Only available for distance average types)
           norm: Order of distance norm as defined in
                numpy.linalg.norm. (Only available for distance average types)
           num_samples: Number of samples on y-axis. (Only available for
                median average type or time on x-axis)