A method to recommend algorithms for software performance benchmarks

A new recommender selects between two steady-state detection methods for benchmark time series.

Researchers developed a recommender to choose between two steady-state detection algorithms. This process is a central step in software performance benchmarking.

The first method is CP-SSD, which identifies steady state through changepoint-based segmentation of the execution trace. The second method is KB-KSSD, which uses a kernel-based approach to identify stable behaviour.

The recommender uses automatically extracted features from benchmark time series. It combines supervised prediction with information from groups of time series with similar behaviour.

The approach was evaluated on 407 labelled benchmark time series. It selected the better detector correctly for 72.97% of the series. This is an absolute improvement of 10.32 percentage points over always selecting the detector that performs best most often, which is 62.65%.

Information from clusters mainly improves interpretability rather than predictive accuracy. Recommendations remain highly stable when the unsupervised representation is learned from the full collection of 5,860 time series.

The results highlight a need for broader labelled data. This is particularly true for execution profiles favouring the changepoint-based detector.

Key facts

  • The recommender correctly selects the better detector for 72.97% of 407 labelled benchmark time series.
  • The method provides an absolute improvement of 10.32 percentage points over the most frequent detector.
  • The approach uses features and information from groups of similar time series.

All sections

Edition